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EXECUTIVE SUMMARY 



For the U.S. Department of Education, Office of Educational Research and Improvement 
(OERI), this study has examined the operations and effects of selected activities of the regional 
educational laboratory program. The ten laboratories conduct applied research and development and 
provide assistance to state and local educators in their regions. In this evaluation, Policy Studies 
Associates collected data from the laboratories and from state and local participants. Across the 
laboratories, we studied activities of the following types: 

• Development coupled with assistance, or the iterative creation and refinement of 
products and processes suitable for wide-scale application, with implementation 
assistance from the laboratory 

• Information products and events; including (1) the provision of research-based 
information in response to requests from policymakers and (2) workshops and 
products that present information to educators 

• Technical assistance to build the capacity of organizations that can in turn play a 
strategic role in school improvement 

• Convening regional groups of educators or policymakers around an agenda of mutual 
interest 

These categories of activities were chosen because they are widespread in the laboratory 
program and because they are different from each other, with each category presenting a different 
combination of opportunities and pitfalls. Collectively, they represent much of what is important in 
the laboratory program-although the activities studied here are not a statistically representative 
sample of laboratory work. 

The methods of data collection varied in depth to correspond to the depth of engagement that 
each participant had had with the laboratory: we conducted site visits to interview a selection of 
participants in long-term development or technical assistance and to understand their local contexts; 
we conducted individual telephone interviews with recipients of tailored products from laboratories; 
and we sent mail surveys to participants in one-time workshops and recipients of products. Again, as 
with the selection of laboratory activities, the selection of participants was large and varied but not 
random or representative. Questions for participants focused on the quality (defined in whatever way 
the participant chose) of products and services received, what the recipient then did, the results that 
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ensued, and any aspects of the laboratory's work that were unique in comparison with the work of 
other comparable organizations. 

These procedures for sampling and data collection equip us to analyze the types of strengths 
and weaknesses found in the work of the regional educational laboratory program but not to assess the 
prevalence of strengths and weaknesses. For example, we can identity the types of changes in student 
performance or behavior reported in some participating sites but cannot say what proportion of sites 
experienced these effects. Despite this caveat, the findings of this study offer some insights into the 
types of strengths and weaknesses characteristically reported by participants in the laboratories' work, 
and these in turn offer a basis for policy directions for the program. 

Cross-Cutting Strengths 

Participants in laboratory activities expressed satisfaction with the experience. Among the 
positive effects they cited were new behaviors among teachers and students; information that they 
passed on to others; and the development of networks for continuing communication. Participants' 
reports pointed to several dimensions on which the laboratories' work was of high quality, which we 
summarize here. 

Laboratories Are Credible Sources of Help 

A key rationale for a public investment in research, development, and assistance is that it 
enables organizations to amass research-based expertise and to apply this expertise impartially* 
without the distortions introduced by commercial self-interest. And, in fact, a number of participants 
explicitly recognized this as a strength of laboratories' work. Laboratories can be trusted to give an 
honest reading of the evidence on policy issues, according to recipients of their policy-oriented 
syntheses. Many participants in development efforts observed with surprise that the laboratory staff 
were willing to revise their products and processes rather than trying to sell them in their existing 
form. Technical assistance in both development and capacity-building activities is reportedly 
distinguished by a genuine openness to understanding the participant's situation in depth and tailoring 
the help accordingly, rather than force-fitting a particular solution to a problem* 
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Laboratories Make Long-Term Commitments to Topics and Partners 



We think thai lhere might have been room tor acceleration in some of the longest 
development activities, bui we are still inclined to agree with laboratory staff that the gradual 
maturing of an activity over at least five years can permit a rich and eclectic mixture of research, 
practice, and evaluation to inform the products and processes developed. The fact that laboratories 
support long-term programs of research, development, and assistance in particular areas can help 
strengthen even those events and products that represent limited time commitments for participants 
and recipients. Other technical-assistance providers, not funded to carry out programs of R&D. may 
have difficulty matching the depth of knowledge brought to bear in the most highly regarded 
laboratory presentations, 

Long-term commitments to partners represent another dimension of strength in the 
laboratories 9 work. Particularly in our category of technical assistance for capacity building, 
participants attributed successes to the laboratories* perseverance in the face of delays and setbacks. 
The repeated interactions with participants in development efforts also contributed to the effectiveness 
of these activities, according to participants. 

Laboratories Are Boundary Spanners 

Most of the activities studied here represent a synthesis of some kind. Most blend research 
knowledge with insights from practice; some bring together different strands of research; several 
bring a policy sense to issues of practice or vice versa. This capacity for synthesis strengthens the 
laboratories as a resource, according to our respondents. Many spoke highly of the practicality of 
laboratories* ideas, contrasting them with more theoretical (and, in the case of policy, more partisan) 
formulations. Practitioners who participated in development efforts told us that they were initially 
surprised to be treated as fellow professionals by the laboratory staff, then went on to describe how 
much they learned from the effort* In another category of activities, convening, the capacity to span 
conventional boundaries clearly contributes to the perceived value of events, which participants often 
described as unique in their freedom from turf issues. 
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Cross-Cutting Weaknesses 



Participants in laboratory work offered relatively few critical comments, but the criticisms 
provide some insight into weaknesses of the laboratories* work. In addition, our analysis enabled us 
to spot missed opportunities. 

Development and Assistance Could Reflect More Engagement with the Field 

Some products, processes, and assistance efforts proffered by laboratories reflect optimistic 
assumptions about the preferences or agendas of practitioners-in other words, they reflect failures ot 
marketing. The public sector sometimes recoils from the idea of marketing, equating it with the 
artificial creation of consumer wants, but in fact effective marketing brings an understanding of real- 
world needs, wants, and interests into the development of products and services. We have seen 
flawed marketing in the laboratories' development and attempted dissemination of many large, 
unwieldy compendia of research findings; in some efforts to enlist educators as volunteer 
disseminators of laboratory processes and products; in a few policymakers' perception that some 
laboratories are only willing to do work that advances a particular agenda; and in the assumption that 
particular organizations strategically situated to assist schools actually share a laboratory's agenda of 
school improvement. 

T -emedy this failing, we urge that laboratory staff spend even more time learning about the 
capacities, agendas, felt needs, and latent needs in schools and other agencies. Conventional needs 
assessment, which is often a compilation of lists of high-priority topics, does not have the necessary 
depth and does not take enough account of the dimension of capacity. Field testing and workshops, 
on the other hand, represent tremendous learning opportunities that should be approached in a spirit 
of open inquiry. This does not mean that laboratories should simply wait for practitioners and 
policymakers to tell them what to do-on the contrary, they must bring an informed and critical 
imagination to the interaction-but tliey must watch and listen. We suggest that laboratories make 
more formal efforts to learn from the conversations that take place during workshops or in trials of 
products and processes. These can be the occasions for inexpensive probes into the field that can help 
inform related efforts in development and assistance. 
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The Targeting of Products and Services Deserves More Attention 



Whether in a development activity that does not seem destined for second-generation 
dissemination at a reduced per-participant cost, or in technical assistance for capacity building that has 
not yet resulted in trickle-down benefits beyond the organization directly helped, laboratories are 
sometimes prone to delivering very good services to very few participants. Not every activity should 
serve a huge volume of participants, of course, but we think the laboratories could do more to press 
for efficiency in their work. Laboratories also should have the flexibility to invest heavily in their 
most promising activities so as to expand their reach. 

In addition to this issue about the number of participants, there is an issue of who the 
participants are. A laboratory is trying to impart knowledge where it will do some good and also 
trying to learn from partners who have a range of perspectives. For activities that involve a serious 
time commitment from participants, this requires the laboratory to strike a balance between addressing 
acute needs and choosing partner organizations that bring commitment and capacity to the working 
relationship. In the relatively few cases where we found disappointment in a working relationship (or 
a short encounter) with a laboratory, the reason was often that the parties brought different 
expectations to the activity. At least for long-term activities, more time should be invested in finding 
interested participants or tailoring the involvement to the participant's interest and commitment 



Laboratories Should Rigorously Scrutinize What Isn't Working 

Some of the field tests and other reviews brought to bear on laboratory activities embody 
systematic designs, formal documentation that captures findings, and commitment to use the 
conclusions-but most of those we studied do not. There is a continual temptation to seek good news 
and favorable ratings; designs do not always capture a range of important effects and issues; and 
developers or assistance providers sometimes forge ahead in the face of what should be clear signals 
to reconsider, a venture. Evaluation and field testing should be opportunities for posing a range of 
choices, asking hard questions, and seeing new possibilities. Laboratory managers and OERI should 
encourage evaluators and program staff to scrutinize the assumptions behind activities as well as the 
mechanics of execution and to radically change or abandon whatever is not working effectively. 
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Communication Within and Across Laboratories Should Increase 



Many of the professionals in laboratories are so engrossed in their own projects that they 
know little about their colleagues' work. This has two unfortunate eftms: they cannot inform 
practitioners about the range of resources available from their own organization; and organizational 
learning is impeded. We have observed some efforts to decrease the isolation within and across 
laboratories, and we urge that these efforts continue and expand. Although there is never enough 
time in the work day for much systematic examination of what has been learned in a project, we think 
that future laboratory activities would benefit from more cross-project and cross-laboratory discussion 
of lessons learned. 

Policy Implications 

OEM's best opportunities for leadership are to be found in framing a Request for Proposals 
(RFP) for laboratory work, establishing performance measures and reporting requirements for the 
program, fostering networking and community building within and across all the programs it funds, 
and framing and communicating a vision of the best possibilities in the work of laboratories. 

Contracted Activities 

The 1985 and 1990 RFPs created categories of activities along functional lines, and one result 
in 1990 was the formal separation of technical assistance from applied R&D, Instead, we would 
recommend encouraging laboratories to organize their work into substantive families of activities, in 
which development, some limited amount of applied research, long-term and short-term assistance, 
and evaluation are organized to inform and support one another. The idea of families of activities 
would build on an existing strength of laboratory work— the spanning of boundaries (among research 
disciplines, between research and practice)-while addressing the existing weakness that some 
activities do not benefit from communication across projects or intense engagement with the field. 
OERI could ask laboratories to demonstrate how sets of interrelated activities will be managed so that 
field experience is scrutinized through the lens of formal inquiry and vice versa. 

A different issue is also likely to arise in the specification of laboratory activities for the next 
contract period. Because laboratories are authorized under the Goals 2000: Educate America Act 
and are charged with some responsibilities under the Improving America's Schools Act. they may 
acquire an obligation to embrace a particular vision of reform content or process. We believe that 
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OERI should strive tor flexibility in this regard, hearing in mind that Goals 2000 is intended as a very 
broad umbrella for statewide— and, indeed, national—conversations about standards and reform 
directions. Harnessing the laboratories to a particular vision of reform, however solidly grounded in 
research that vision may seem, wouid seriously undercut the neutrality and hence the credibility that 
they now bring to policy discussions in their regions. 

Performance Measures and Reporting Requirements 

Counting participants in laboratory events provides only one view of the program's 
accomplishments. Although very low numbers may show a disturbing lack of energy or connection 
with the field, very high numbers may show superficiality in the program of work. It will clearly be 
important to balance measures of depth of engagement with measures of breadth of coverage. 

Our data indicate that "customer surveys" wilt result in high ratings for the work of 
laboratories. To the extent possible, it will be important to probe beyond measures of satisfaction 
with the form or content of laboratory services and products. In particular, it will be interesting and 
useful to ask participants how they use what they have learned from laboratories-and to ask them 
what, if anything* the laboratories have learned from them. We would also advocate a real press for 
information on behavioral changes and, especially, on student effects that are plausibly attributable to 
the work of laboratories. 

To combat the defensiveness now found in some evaluation and field testing in the laboratory 
program, OERI should send a clear message that it expects and will tolerate some reports of failure. 
Laburatories should be publicly praised for identifying their mistakes -provided that they show 
evidence that they are applying the lessons learned. 

Networking and Community Building 

We have observed that the laboratories' work would benefit from more opportunities to 
identify and apply the lessons that emerge from related work within and across organizations. OERI 
should require and facilitate communication among laboratories at the program level, especially 
among staff members who are working on substantively related projects. Laboratory evaluators 
should continue to meet as a group. 
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New legislation requires the laboratories to collaborate with other federal programs, including 
the institutes, the National Diffusion Network, and the comprehensive regional assistance centers. 
OER1 should recognize that collaboration is time consuming and therefore expensive, especially when 
it is not very actively promoted, facilitated, and modeled by federal officials. A useful first step, 
then, would be for the laboratory team in OERI to build strong lines of communication with these 
other programs. 



Vision 

Federal policymakers, if they have heard of regional educational laboratories, typically know 
little about the work of this program. In this evaluation, we have tried to contribute to a program- 
wide clarification of what laboratories, in the current phrase, know and can do. Based in part on this 
information-and in communication with boards, managers, and professionals in the laboratory 
program-OERl should formulate a vision of high aspirations for this program, then hold the 
laboratories accountable for realizing and further strengthening the vision. This would differ from a 
minimalist policy of expecting success in conducting discrete projects, technical-assistance events, and 
regional meetings. 
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L INTRODUCTION 



Ten regional educational laboratories receive support from the U.S. Department of Education, 
administered by the Office of Educational Research and Improvement (OER1), to provide assistance 
and applied research and development (R&D) for specific geographic regions of the country. At 
present, most of the work of laboratories can be described with reference to the two largest tasks in 
the 1990 request for proposals (RFP): 

• Assistance : Working with educational decisionmakers and practitioners in the region 
io help improve educational outcomes for at-risk and other students through such 
activities as technical assistance, staff development, evaluation, dissemination, and 
meetings that convene educators and policymakers 

• Applied R&D: Contributing to the regional and national knowledge base through 
such activities as developing organizational or instructional models, testing or 
demonstrating model programs, conducting studies, and producing R&D syntheses 

Some laboratories make a distinction between their assistance activities and their applied R&D 
activities, while others meld the two tasks within broader-purpose activities. 

Originally authorized by the Elementary and Secondary Education Act of 1965, laboratories 
were reauthorized in 1994 as part of the Goals 2000: Educate America Act. Title DC of that act 
establishes an infrastructure for R&D and technical assistance in OERI, including institutes 
specializing in high-priority areas of research as well as a variety of programs— of which the 
laboratory program is the largest-that operate in more direct contact with practitioners. The law 
assigns a lengthy list of functions to laboratories, most of them variations on the themes of technical 
assistance and development. 

This evaluation has been a three-year effort, launched within the first year of the laboratories' 
1990-95 contract period. Conducted by Policy Studies Associates under a contract with OERI, this 
evaluation is intended to describe several key aspects of laboratory operations and the effects of 
selected laboratory activities, to identify strengths and weaknesses in the laboratory program's overall 
performance, and to suggest ways of improving the program. The evaluation was designed to help 
inform development of the next RFP for laboratories, which will set a framework for their operations 
from 1995 to 2000. The evaluation has also sought to provide ideas that the laboratories could 
voluntarily implement in order to strengthen their operations and effectiveness. We were not asked to 
render an overall verdict on the worth of the program, and we have not taken it upon ourselves to do 
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so. Our purpose has been to assess important features of' its current operations with a view to 
program and policy recommendations for future operations. 

This final report addresses the question of the operations and effects of selected laboratory 
activities. It is based on a purposive sample of activities across the ten laboratories. We do not claim 
that these activities are representative of all work conducted under the laboratories' contracts. 
However, we did make a very serious effort to capture the important categories of laboratory 
activities and to illustrate successful and unsuccessful experiences. 

The term "activity" deserves some definition. Most of our resources went into studying 
whole projects that have been substantial in scope and duration and that laboratory staff would 
identify as integral units of their work. However, we also studied some very small-scale activities- 
ones that required limited time commitments from both laboratory staff and external participants. 
Examples of the latter type include an array of efforts to provide research-based information to policy 
audiences on request. And in a few cases where a program of work has had a number of interrelated 
parts, we somewhat arbitrarily isolated one part for examination. An example is the work of one 
laboratory with an urban school district in connection with Chapter 1 schoolwide projects; this activity 
is tied in practice to several kinds of assistance and research that have taken place at all levels from 
state policy to school buildings. 

We chose activities for study after a process of preUminary data collection and consultation 
with OERI and the laboratories. The laboratory executive directors, evaluators, and program 
r, onagers cooperated with our effort to capture a range of activity types and to learn from failures as 
well as successes. Inevitably, some cooperated more wholeheartedly than others; thus, some parts of 
the evidence for this report reflect best-case scenarios while other parts contain harsher reality. It is 
our sense, though, that we have been able to see the types of limitations associated with each major 
category of laboratory work and to draw lessons about overall, cross-cutting weaknesses. 

The main reason we trust our evidence as much as we do is that it comes from the field-from 
school staff, other service providers, administrators, and policymakers who have participated in 
laboratory activities or received laboratory products. We used different methods of data collection, 
varying in depth to correspond to the depth of engagement that each participant had had with the 
laboratory: we conducted site visits to interview a selection of participants in long-term development 
or technical assistance and to understand their local contexts; we conducted individual telephone 
interviews with recipients of tailored products from laboratories; and we sent mail surveys to 
participants in one-time workshops and recipients of products. Again, as with the selection of 
laboratory activities, the selection of participants was large and varied but not random or 
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representative. 1 Our questions for participants fiicused on the quality (defined in whatever way the 
participant chose) of products and services received, what the recipient then did, the results that 
ensued, and any aspects of the laboratory's work that were unique in comparison with the work of 
other comparable organizations. 

We have analyzed these data across activities within four broad categories that seemed to us 
to capture important areas of laboratory endeavors-and that, as we analyzed our findings, turned out 
to hang together in their characteristic types of results and limitations. Judgment calls abound in our 
assignment of activities to categories. Thus, some activities that the laboratories classify as 
development are called technical assistance here because nothing has as yet been developed for 
broader application; other activities that have resulted in the development of products and processes 
are classified as short-term information events because that is what they are for participants. Readers 
should bear in mind that we do not ascribe policy significance to the categortes-we are not trying to 
say that some are good bets for public investment and others are not-but instead are simply trying to 
combine activities in ways that would bring some interesting similarities to light in our cross-case 
analysis. 

A final caveat about our evidence and findings is especially important. If the activities 
studied here are not statistically representative of the laboratory program as a whole, still less are they 
representative 01 the work of any single laboratory. For most laboratories, we studied the delivery of 
several policy information produces; for every laboratory, we studied two or three additional 
activities. Federal policy treats the laboratories as a single program with shared authorizing language, 
an appropriation, an RFP, and occasional group events hosted by OERL Policy is not made for 
individual institutions, and our data base does not support conclusions at the level of institutions; for 
both of these reasons, we do not offer any, and none should be inferred from the text that follows. 



1 We do. however, present frequencies drawn from the surveys of workshop participants and 
product recipients, where we gathered complete lists and drew samples from them (a random sample 
from the longer lists, the whole list from the shorter ones). Because the surveys had many common 
items across activities, we were able to analyze the data in the aggregate and thus to compile a 
broad-though not strictly representative-picture of the response of participant* to this category of 
laboratory activities. 
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II. DEVELOPMENT COUPLED WITH TECHNICAL ASSISTANCE 



Among the large-scale, flagship activities of laboratories arc those that encompass 
development and technical assistance. Typically conducted over a period of several years, these 
activities feature the iterative development of research-based mode] practices for teachers, 
administrators, or entire organizations. At least in the early iterations, and sometimes continually, the 
laboratory modifies its products and processes on the basis of field experience. Down the road, 
products continue to be accompanied by technical assistance, usually from the laboratory. 

Activities Studied 

We briefly describe here the 1 1 activities considered in this chapter, identifying their general 
puiposes, the products and processes developed, the interactions with participants, and any changes 
made in their designs over time. These descriptions are followed by a summary of our rationale for 
grouping the activities in this analysis; we then present and discuss our findings from participants. 

A ppalachla Educational laboratory's (AEL'sl QUILT 

QUILT (an acronym for Questioning and Understanding to Improve Learning and Thinking) 
has twin aims: to help teachers improve the quality of their questioning in order to create a more 
reflective classroom environment and enhance student thinking; and to demonstrate a staff- 
development model capable of changing teachers' classroom behavior. The research base, 
accordingly, includes both research on teacher questioning and research on effective techniques of 
staff development. With QUILT, AEL has developed and tested a train -the-trainer model in which 
laboratory staff train schooi-bas.<xi teams of teachers and administrators, who then train and support 
their colleagues in learning and applying techniques of higher-order classroom questioning. These 
questioning techniques include, ft r example, wait time and asking questions at all intellectual levels. 
QUILT offers copious materials for trainers and trainees alike; thus the direct involvement of 
laboratory staff members, which accompanied the program's development, does not need to continue. 

As of spring 1994, 130 schools and 2,500 teachers had received training. The early years of 
QUILT included a formal field test in which laboratory evaluators compared the impact of three 
variations of QUILT training: three days of inservicc plus seven 90~mii> »te collegiums and seven 
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peer observations over the course of the school year; the initial three days with no collegiums or pee/ 
observations; and only a three-hour workshop. Based on the results of this field test, the first and 
most elaborate design was the only one successful in changing teacher behavior; it thus became the 
final QUILT model. 



AEL's Study Groups 

In AEL's Classroom Instruction program, most Study Groups are small groups of six to ten 
teachers who have volunteered to meet regularly over a school year to explore a topic in depth. 
(Another laboratory program has also conducted Study Groups, but we collected data on those 
involving teachers.) Over the past two school years, the Study Groups have varied in size from five 
to 21 teachers, plus representatives of state education agencies and professional associations; they 
average three or four meetings over the school year. Topics for 1993-94 included community service 
and service learning in Kentucky, early childhood transitions in West Virginia, assessment in 
Tennessee, and alternative assessment and integrated instruction in Virginia. Most often, the 
participating teachers read the literature, gather data (usually through a group-constructed survey), 
and write a final report* Less often, they explore an innovative practice in their own classroom or 
school, keep up with current research on the practice, and write a final report about their experiences 
and their effective innovations. A Virginia Study Group that began its work in 1991 continued for 
two years in order to complete its work on alternative assessment; the 1993-94 group then built on 
that work by experimenting with new practices in the class r oom and adding integrated instruction to 
the investigation. 

About 40 teachers per year participate in Study Groups-37 in 1992-93 and 43 in 1993-94. 
The total for this contract period to date is about 200. Learning from their experience since 1986-87, 
laboratory staff have codified a set of guidelines for structuring Study Groups: group size should be 
small; groups should select an issue that is both important and feasible to address; members should 
develop their own processes and products; costs should be low and should be shared by AEL, 
participating agencies or associations, the individual participants, and their employers; AEL should 
work with professional associations to facilitate group organization and functioning and to disseminate 
the products. 
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Far West Laboratory's (FWL's) Case Methods 



Cases are narratives drafted and revised by classroom teachers with assistance and guidance 
from researchers. In a few pages, a case describes the context, portrays one or more classroom 
events (including examples of student work and dialogue), and reflects on these events. Assembled • 
into books that address a common set ot themes, cases form the basis tbr-typically-monthly two- 
hour meetings of teacher groups, numbering from six to fifteen, facilitated by teacher educators or 
teachers. Thus, development in this activity includes both the development and refinement of the 
casebooks themselves and, over time, the development and refinement of techniques for facilitating 
discussion groups. Laboratory staff members have worked directly with the groups and, thus, 
provide ongoing assistance with implementing Case Methods. Casebooks developed to date by the 
laboratory deal with mathematics in grades 4-8, mentor teachers, teacher interns, and diversity. 

North Central Regional Educational Laboratory's (NCREL'sl Strategic Reading Program 

The Strategic Reading Program (SRP) provides materials, staff development, and ongoing 
assistance designed to equip teachers to use five strategies in the teaching and learning of reading: 
prior knowledge, word meaning, text structure, inferencing, and metacognition. Constnictrvist 
cognitive psychology is the basis for these strategies, which laboratory staff assembled into training 
materials in the 1980s. Each participating school in the sites other than Detroit is required to send a 
team to the initial training session, consisting of the principal, reading or media specialist, and two or 
three teachers; upon returning to their school, teachers practice the strategies in their classrooms and 
all team members work together to help each other and to train other teachers. The teams return to 
NCREL in tL? next two summers for additional three-day training institutes. The Detroit schools 
have a different training process: most do not send teams to the summer institutes but instead receive 
their own series of workshops from central-office reading consultants and laboratory staff. The 
ongoing support available for all schools includes monthly newsletters and monthly audio-conferences; 
in addition, the rural schools receive two visits per year from field agents. 

All told, 72 school teams (from 35 rural schools and 37 urban schools, 23 of them in Detroit) 
have attended at least one summer institute; most have completed the three-year training cycle. Each 
team includes four or five people. In addition to the core teams, a large number of teachers 
participate in the onsite workshops and other training activities provided by the SRP teams and 
laboratory staff in the schools. Evaluation has addressed the training process, the use of SRP 
strategies in classrooms, and student outcomes. Over time, the most notable change in the program is 
the deemphasis of technology; despite the original intention of using distance learning for teachers in 
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remote rural sites, (he participating schools lacked the capacity to use modems; and e-mail, and most 
teachers find the audio-conferences hy telephone to he a more convenient and comfortable way to 
remain in touch. 

Northwest Regional Educational Laboratory's (NWREL's) Creatine the Future 

Creating the Future builds and expands on an older laboratory project Onward to Excellence, 
which developed model processes based on effective-schools research and provided technical 
assistance in applying these processes. Similarly, Creating the Future includes research-based 
processes and continuing assistance, but it operates in a wider arena than the individual school: its 
target is systemic change in districts and communities, focused on strategic planning and data use to 
improve student learning. 

Participating districts, of which there have been 19 since 1990, form local leadership teams, 
each of which represents a "vertical slice" of the district including, for example, central office 
administrators, building staff, school board members, community members, and students. Laboratory 
staff assist the leadership teams for at least two years, with one or two staff members meeting with 
each team six or seven times per year for a few hours or a day. Initially, they help the team organize 
itself; they provide workshops on getting started, profiling, developing shared meaning, formulating 
statements of student outcomes, and setting priorities; they also provide another session on profiling 
and two workshops on assessment. They work with the team to analyze the information gathered in 
the district and to work through the process of setting districtwide learning goals for students. 
Troubleshooting meetings can be called by either the laboratory or the site, and laboratory staff are 
always available for telephone consultation. They give the sites voluminous materials designed or 
selected for Creating the Future: copies of articles and reports on school reform, and overheads and 
other guides for the leadership team to use in its work in the district. Finally, laboratory staff are 
documenting the process in each district. 

NWREL's Integrating Education and Human Services 

The laboratory is working with pilot sites to develop and field test tools and materials that will 
provide information, awareness, and assistance for local collaborations in the integration of education 
and human services for families and children. Drawn from research and field experience in service 
integration, the conceptual framework for this project centers on six elements of successful 
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integration. The products, which are still under development, include diagnostic tools and awareness 

workshops. 

Participants include representatives of social service agencies, state and local education 
agencies, and collaborative entities planning or delivering services. In its current, developmental 
phase, the project involves relatively intensive work with five field-test sites as the laboratory 
develops tools for assistance such as community profile instruments, which are diagnostic tools that 
help define issues and needs for changes or training within local collaborations. For example, the 
laboratory worked closely with these sites in making presentations, administering the profile, 
analyzing it and presenting feedback, and planning or conducting training. Representatives of the 
field test sites also constitute an advisory board for this program. In addition, awareness workshops 
of two to eight hours in length have been developed and presented to at least another 14 sites. 

Pacific Region Educational Laboratory's (PREL'sl PRESS 

Pacific Region Effective and Successful Schools (PRESS) is a process for developing 
partnerships among all school staff, families, the community and cultural leaders, and the district or 
entity education department staff. Originally based on two NWREL activities, Onward to Excellence 
and Successful Schools, PRESS has evolved away from a central focus on the effective-schools 
research base and now emphasizes community involvement and cultural relevance. It has seven 
modules: awareness, profiling and cultural impact research, goal setting, action planning, 
implementation, monitoring and assessment, and evaluation and renewal. Each module includes a 
trainer's guide outlining content and process, handouts and overheads, and supporting materials with 
background information for trainers and school leaders. 

The laboratory has recruited four or five trainers from each entity, representing a variety of 
roles (teachers, principals, curriculum specialists, and other central-office staff). Development of 
PRESS has been a joint effort of the trainers and laboratory staff, and it is continuing; not all modules 
are complete. The trainers work with interested schools and community leaders to initiate and 
facilitate the PRESS process. Among trainers who work in central offices, PRESS is often part of 
their job; for those who are teachers or principals, it is an additional, voluntary responsibility. The 
laboratory maintains contact with the trainers by means of two meetings per year, monthly conference 
calls via satellite, and a quarterly newsletter for participating sites. PRESS participants range from an 
entire village to a number of schools and their communities to one school and its communities. 
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Partner Schools of the Regional Laboratory for Educational Improvement of the Northeast and 
Islands (NE/ IS) 



The Partners are a confederation of 40 elementary schools that have made a commitment to 
engage in a transformation process. Each school has a contact person in the laboratory, is invited to 
participate in an annual conference, sends a representative (preferably the principal) to three or four 
steering-committee meetings per year, and sends representatives to "working parties" on topics such 
as assessment or community development. The schools also receive newsletters and computer 
communication and are invited to laboratory conferences. The topics of the overall Partners effort are 
wide ranging, including the change process (with specific attention to teaming and group decision 
making); learner-centered, constnictivist approaches to teaching and learning; assessment; the urban 
learner; multi-age grouping, inclusion, and other strategies based on the belief that all children can 
learn; integration of education and social services; and, in general, the exploration of new 
organizational structures and arrangements in support of the learning environment. 

This activity was designed to enable school and laboratory staff to learn about the change 
process together. Documentation is extensive and is intended to yield new knowledge of both short- 
term and long-term utility: for example, there is continual retooling of strategies within the Partners 
program; in addition, laboratory staff are seeking to add to the overall knowledge base on school 
change. Since the inception of this program, the laboratory has begun to provide a greater amount of 
assistance to participating schools; although this is still a nondirecttve effort relative to most 
laboratory work nationwide, the laboratory staff has stepped forward from what was originally 
conceived as primarily the role of observer. 

NE/IS's Teacher Induction Working Group and Mentoring Guidebook 

This three-stage activity encompassed the initial development of a guidebook for mentor 
teachers by SEA staff members with assistance from the laboratory; the refinement of the guidebook 
in eight pilot sites; and interaction among the pilot sites and states in the Regional Mentoring 
Network. For the Working Group of eight SEA staff members, who have responsibilities in 
professional development, developing the guidebook involved compiling information and resources, 
working together in monthly two-day meetings, and completing their writing assignments between 
meetings. In this process, laboratory staff provided materials, reviewed literature, facilitated the 
meetings, and made themselves available for assistance or input at any time. The eight pilot sites, 
selected from a pool nominated by the Working Group, met three times during 1992-93 to receive 
training and help revise the guidebook on the basis of trying out its activities. Finally, the Regional 
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Mentoring Network, with the pilot sites as its nucleus, will continue this working relationship and 
extend it to a broader membership. 

Research for Better Schools* (RBS's) Applied Research on School Restructuring 

This activity, unlike others discussed in this chapter, cannot be considered to have 
development and technical assistance as its primary aims and methods. Instead, the investigators 
emphasize that they a/e conducting research on school restructuring and that the primary audience for 
their work is the research community. Nevertheless, Laboratory staff do see themselves as a resource 
for participating schools; they are trying to structure their work so that the schools will he able to use 
the data; and products Mich as surveys may have future utility in other schools. 

Laboratory staff are studying the effects of school restructuring on students' school 
experiences in five middle schools and one elementary school. Initially, the laboratory put together 
structured interview questions and interviewed teachers and then students. At this point in the 
project, the focus is on students-results of restructuring, day-to-day activities, and contextual factors, 
Working with the schools, laboratory staff develop paper-and-pencil surveys; teachers administer 
surveys to students; the laboratory analyzes the data and provides feedback to school faculty and 
administrators, who are free to request follow-up consultations if they want to. 

SouthEastem Regional Vision for Education's (SERVE'sl Formative Teacher Evaluation 

Formative Teacher Evaluation is intended to take the place of summative techniques in 
evaluating the performance of experienced teachers. While those involved in its development and 
pilot testing believe that conventional checklists may have value for beginning teachers, they observe 
that districts use these checklists with all teachers, even though they are not required to do so. Thus, 
the developers are seeking to move school systems toward the use of formative evaluation among 
experienced teachers. Using this technique, teachers can identify skills they want to develop- 
especially skills connected to a reform agenda, such as higher-level questioning-and use formative 
evaluation as a means of obtaining feedback from administrators, colleagues, or (through video) 
themselves. 

The product of the laboratory's development effort is a handbook presenting the rationale for 
Formative Teacher Evaluation, contrasting formative and summative approaches, and documenting the 
specific techniques used in three pilot districts. Laboratory staff worked closely with these districts 
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during the pilot phase, presenting workshops and following up with individual help. The districts, in 
turn, evaluated their work (using their own evaluation designs) and reviewed drafts of the handhook. 
One- or two-day workshops that introduce new districts to the idea of Formative Teacher Evaluation' 
are conducted by central staff and teachers from participating districts, with help from lahoratory 
staff. 

Common Elements in These Activities 

The activities grouped for discussion in this chapter share a number of features. They are 
large, long-term, and generally high-profile effbrts-unlike the more ad hoc or short-term work that 
laboratories engage in. And they employ two key methods: development and technical assistance. 

Development 

Over a period of years in these activities. Laboratory staff develop something-that is, they 
refine a set of products and processes designed for use in a relatively standard way with many 
participants over time. A "rollout" is very much expected. The fruits of development are especially 
easy to identity in projects that focus on teachers* classroom behavior: QUILT and Strategic Reading 
both encompass specific sets of skills for teachers to use; and they also feature tested procedures for 
imparting these skills. Two other professional-development activities reflect the development of 
standard, tested procedures for teacher learning: Study Groups and Case Methods reflect the field- 
based refinement of principles and procedures for group work. (Each of these activities has also 
resulted in the development of tangible products-the research syntheses written hy the Study Groups 
and the casebooks used as the basis for case discussions -but in this analysis we emphasize the 
development of the method of teacher learning.) 

Development and assistance are also brought to bear on larger organizational units than the 
single participating teacher or classroom. Strategic Reading, despite its emphasis on classroom 
behavior, is organized around the grade level or school as the participating unit. Partner Schools 
focuses on school buildings. Formative Teacher Evaluation is intended for adoption by whole 
districts or at least schools. Creating the Future and Integrating Education and Human Services reach 
beyond school systems to include communities among those to be involved in implementation. 

Across all these activities, different phases of the development cycle can be observed. In 
some of the activities studied, the development phase is essentially complete (although further 
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refinements of products and processes are always a possibility). Again, QUILT and Strategic Reading 
are examples. The Mentoring Guidebook is also complete, as is the handbook on Formative Teacher 
Evaluation. Other activities are now in their development phase. In Integrating Education and 
Human Services, laboratory staff are working with partners in field-test sites to revamp their draft 
products. The Partner Schools activity is part of a larger program that is in the early phase of a very 
long timeline, and the work with the partner sites is largely exploratory at this point; we discuss this 
activity in this chapter hecause development is expected to ensue. 

Technical Assistance 

Although some descriptions of laboratory work, notably the 1990 RFP, identify development 
and technical assistance as separate spheres of activity, these activities strikingly illustrate how they 
are intertwined in practice. In the activities described here, laboratory staff have been in the field 
working with virtually every user of the products and processes they have developed. Indeed, for 
most of the sites that have implemented any of the products and processes discussed here, technical 
assistance from the laboratory j& the activity. This interaction is not confined to the early stages of 
development and field testing. Instead, it persists throughout the life of an activity. 

Only rarely do laboratory staff plan to take themselves out of the loop of adoption and 
implementation altogether. QUILT is the most notable exception, with materials and a training-of- 
trainers package designed to equip local school staff to implement the product without continuing hc'.p 
from the laboratory. However, other activities also reflect the idea that some use can occur without a 
continuing laboratory presence. Strategic Reading expects the core teams to impart the philosophy 
and skills to their colleagues in participating schools. Case discussions do not have to be facilitated 
by laboratory staff, although the teachers and staff developers who now facilitate discussions have had 
several years of direct work with the laboratory. The laboratory's continuing role in sites adopting 
and using Formative Teacher Evaluation has not been defined, but it is expected to diminish over time 
as a larger corps of field-based practitioners gains more experience and expertise with this method. 
Similarly, both NWREL activities-Creating the Future and Integrating Education and Human 
Services-are designed to use cadres of field-based trainers someday. 

Outliers and Overlaps 

Readers should hear in mind that the categories in which we have placed I ah oratory activities 
are our own analytic construct; they reflect our observation of the principal clusters of laboratory 
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work nationwide, rather than any design principles established by OERI or the organizations 
themselves. Thus, the categories are far from neat, and no negative conclusions about lahoratory 
work should be inferred from their messiness. Here, we mention the features of the activity that is 
the most awkward analytic tit in this category, and briefly descrihe areas of overlap with other 
categories. 

RBS's applied research on restructuring in six schools does not emphasize the development of 
products and processes for use in other settings, but it illustrates the fine line hetween applied 
research on the one hand and development on the other. Both endeavors draw on a base of existing 
research, and hoth generate something tangible, of wider utility. In the case of applied research -- 
unlike the other activities described and analyzed in this chapter --the product is a set of research 
findings and methods, aimed primarily at the research community. In development work, the product 
would he a collection of skills and procedures tor application in practice settings (or, conceivably, in 
a policy setting). However, this applied research project shares another important feature with the 
development work discussed here: rather than using the participating schools merely as data sources, 
the lahoratory staff have cultivated a continuing relationship in which they provide at least some 
assistance to the schools. Indeed, the laboratory staff are not without amhiva\ence on the issue of 
purpose. One staff member addressed the question of whether to classify this activity as technical 
assistance in this way: 

We struggle with that issuo all the time. We try to make the data more interesting to teachers 
and administrators. But we come to a point when we say enough is enough-the data is for a 
larger research audience. 

The next chapter of this report discusses workshops and products that impart information to 
participants and recipients. These do reflect development efforts hy laboratories, and in some cases 
th- development effort has been lahor intensive. We have grouped them together in a separate 
chapter rather than Deluding them here because, from the participant's perspective, what the 
laboratory offers is * one-time event or single product rather than repeated occasions for assignee. 

Finally, this category has a good deal in common with our category of technical assistance to 
huild the capacity of strategically situated organizations. The key difference, in our view, is that the 
activities discussed in this chapter are strongly geared toward the creation of products or model 
processes for eventual rollout. In the other category are more ad hoc efforts, where a primary aim is 
to help another agency do its work more effectively, and a replication of the laboratory's or other 
agency's procedures on a wider scak is unlikely. 
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Results 



The laboratories' activities that comhine development and assistance have had results uf tour 
general types, according to field reports: research-hased products and processes are created; 
participants receive technical assistance that they consider to he of high quality; hehavioral changes 
take place in implementing sites; and other effects (differing across activities) are found among 
students, teachers, and other participants. Each type of results has policy significance. 

Products and Processes Are Develop ed 

Among the aims of federal R&D policy is the creation of user-friendly vehicles for putting 
research findings to work in practice. Development is an elaborate but generally effective way of 
doing this. In contrast to presentations or publications that digest research findings for a practitioner 
audience, development projects go several steps farther. They use research findings as a basis for 
designing specific actions thai practitioners can lake; these are codified in the form of products (which 
may be quite prescriptive, as in the case of scripts for teachers or trainers, or more general); and the 
products undergo field testing and revision. At the end of this sequence, the result is a tested process 
that other practitioners can execute. To say that these activities have resulted in products and 
processes is not just an exercise in defining a category of activities; some laboratory efforts are ' 
intended to develop products and processes hut in fact never arrive at that point. 

They are based an research. In most of the development activities we studied, as the 
following examples illustrate, laboratory staff drew on research to formulate procedures that 
practitioners could carry out. The research base gives a reason to expect that the procedures will 
work effectively. 



Integrating Education and Human Services pulls together existing research on the 
delivery of comprehensive services to families and children. The conceptual 
framework began with a dozen key concepts, which have since been boiled down to 
six. These six elements are the basis for a diagnostic instrument that participating 
sites apply to their own work. 

The Mentoring Guidehook codifies research on mentoring to arrive at guidance for 
specific training and mentoring procedures. (In addition, each chapter of the 
guidebook begins with a review of research on the particular topic addressed in that 
chapter, along with a resource list for further reading.) 
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• Strategic Reading draws on research in staff development (including reflection on 
practice, peer coaching, and staff development leadership) and in the cognitive 
processes involved in reading as a goal-directed, strategic process of creating 
meaning. According to a lahoratory staff member, in the early 1980s "a lot of people 
were talking ahout strategic learning, hut no one had put the pieces together." 

• Similarly, QUILT unites research on staff development and research on teacher 
questioning. The staff-development literature informed the decision to offer fol low-up 
collegiums and peer observations during the school year; research on pedagogy 
informed the aim of increasing teachers' use of higher-level questions and wait time. 

• Case Methods draw on cognitive flexibility theory, which holds that in an ill- 
structured domain (like teaching) one learns "by encountering the same issues in 
different contexts and from different perspectives," according to a staff member. The 
mathematics cases apply Lee Shulman's concept of pedagogical content knowledge. 

• Partner Schools is an activity steeped in research, although the laboratory is opposed 
to the idea of translating research into firm prescriptions for practice. The work 
draws on the literature on organizational change as well as that on learner-centered 
approaches to teaching and learning, with antecedents that go hack to Dewey and 
Piaget. 

• Creating the Future incorporates concepts of systemic change, ?nd a senior staff 
member says of the process, "From day one we are modeling a constructive learning 
process [for participants].* It also draws on theories of outcome-hased education, hut 
because of the opposition sparked by that movement, these ideas are now being 
applied only very selectively in the program. 

These examples illustrate a characteristic common to many Laboratory activities: they pull 
together different strands of research. In university settings, it is uncommon for research on teaching 
and learning to be knitted together with research on staff development or organizational change in the 
design of a single project. For laboratories, however, this is a customary way of working. 
Individual senior professionals in laboratories can pursue a broad range of interests, and they can 
work in teams with colleagues whose disciplinary backgrounds are different. The result, when the 
process works effectively, is the development of products and processes that apply multiple 
knowledge bases. Thus, laboratories are equipped to work in a way that contrasts with some other 
parts of the R&D universe, in which researchers primarily disseminate the findings of particular 
studies. 

Although the leaders of all these development projects cite a research base when asked, their 
remarks reveal three different kinds of orientations to the world of research. These may intermingle; 
no activity reflects a pure type of any one orientation, and the orientation may shift over time within a 
particular activity. One orientation is almost a cursory wave of the hand: a developer rattles off a 
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general principle that is embodied in the products and processes the laboratory has developed, 
something along the lines of, "parent involvement is important in education, 11 to choose a hypothetical 
example. (This can sound like mere sloganeering, hut, in fairness, it may simply reflect the kind of 
shorthand formulations used among professionals in any field.) Another is the mining of the research 
literature for more specific, practical prescription*, which the laboratory's products codify in "how* 
to" form for practitioners. Both of these orientations are consistent with conventional policy 
formulations of the role of laboratories. 

A third orientation views research in a less static light, as a process of inquiry that is 
continually evoiving and to which laboratories and their partners in the field are themselves 
contributors. This intellectually ambitious orientation has its pros and cons. On the positive side, in 
our view, it gets inquiry off the shelf and into the schools in a more dynamic way than conventional 
R&D policy has envisioned, with the result that a great deal more problem-solving energy can he 
unleashed on the creation of new knowledge (Turnbull, 1994). The downside is that an activity that 
is continually evolving may remain so amorphous as to leave practitioners dissatisfied, as we will 
discuss in a later section of this chapter. 

They have teen field tested. These activities also reflect several field trials of the procedures 
originally developed. At a minimum, operational details have been ironed out on the basis of field 
experience. At the more elaborate end of the spectrum, formative evaluation has allowed the 
laboratory to reshape an activity in fairly basic ways. 

QUILT stands out here as an example of formal field testing, carefully designed to answer a 
specific question: how extensive should the staff development he? The developers compared the 
hehavioral changes found under three different conditions, and they learned that collegiums and peer 
observations during the school year were critical in bringing about actual changes in teaching. Thus, 
they abandoned the designs in which training took only three days or three hours. 

The level of formality and the 'ype of documentation associated with field testing vary across 
activities. This can be Illustrated by the three activities that offer staff development on a range of 
topics: Study Groups, Formative Teacher Evaluation, and Case Methods. Study Groups reflect the 
refinement, through field experience, of a set of standard operating procedures by which participants 
can become acquainted with a body of research in some depth. In the early years of Study Groups, 
laboratory staff members learned several lessons, which they codified in a 1987 report: for example, 
that groups should be no larger than about ten participants, that they should develop their own 
processes and products, that costs should be shared, and that laboratory staff should facilitate each 
group's organization and functioning. At this point, the procedures tor Study Groups are well 
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established and not suhject to much further refinement, although each group provides evaluative 
teedhack on its experience. 

Formative Teacher Evaluation is a process in which field experience has helped to give more 
concrete form to "a really open-ended" idea, as a project staff memher descrihes it. Lahoratory staff 
emphasize with some pride: "We don't have a canned Formative Teacher Evaluation plan. We 
provide some options and some methods, and they [in the adopting district) develop a plan. " The 
handhook developed hy the team, with considerable input from the first three districts using 
Formative Teacher Evaluation, describes the operational specifics of the plans developed and carried 
out hy these districts; it is designed to include the theory behind the process but to emphasize a 
practical experience hase. Nevertheless, the concrete shape given to Formative Teacher Evaluation si> 
far is not the end of the development road, according to lahoratory staff* They expect a further 
evolution in these and other districts. And, unlike some other laboratory products, this one is 
apparently expected to remain open-ended forever; there is no press to use field experience as a way 
of identifying more effective procedures and then to recommend or mandate their use. In this 
activity, then, development is almost synonymous with local implementation, which is expected to 
reflect unique local priorities and preferences. 

The developers of Case Methods are contributors to the published literature on teacher 
preparation and professional development, and their field experience with the writing and discussion 
of cases has had something of the flavor of applied research. These developers-like some of their 
colleagues in the national laboratory program but unlike others-are part of an ongoing conversation 
with researchers in universities and other laboratories, and their understanding of what works in field 
trials is informed by this conversation. Research on professional development, such as the literature 
on organizational and individual change that supports critical and collaborative cultures among 
teachers, gives them criteria by which to assess their accomplishments and refine their methods. ' 

Whenever a laboratory develops something, the field trials inform development in some way. 
As these examples illustrate, activities differ in the mix of practical and theoretical criteria applied in 
the course of development. They also differ in the extent to which development is expected to reach 
a clear endpoint, and in the amount of local variation expected to persist in implementation. 

Inevitably, Meld testing will bring to light some flaws in a product or process that in 
retrospect appear obvious. Criticizing these flaws after the fact would be unfair-except when they 
recur time after time. We cannot help being struck hy one recurring flaw: the tendency of 
laboratories to develop long, user-unfriendly documents. Paring down the length of draft materials 
was part of the revision process in Strategic Reading, Integrating Education and Human Services. 
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Applied Research on School Restructuring, and Formative Teacher Evaluation. Participants say that 
revisions should have been more extensive in the Mentoring Guidebook (which runs to some 800 
pages) and Creating the Future (which has generated seven three-inch hinders). In a district that has 
participated in hoth Creating the Future and its closely related predecessor. Onward to Excellence 
(OTE), a slogan we heard from several staff members was, "Kill a tree for OTE." These criticisms 
of laboratory products underscore the necessity of a reality check for R&D products that are also 
contractual deliverables to a government agency. OERI seldom if ever presses laboratories to 
streamline their written products; field testing is the only source of pressure for conciseness. 

Technical Assistance Is Provided Through Partnerships 

A characteristic of laboratory work that stands out in most of these examples of development 
is the partnership with field sites. Participants at the field sites were virtually unanimous in praising 
the laboratories tor recognizing practitioners' capacity to contribute, as professionals, to the 
development process. The comment of a participant in Integrating Education and Human Services 
captures this response nicely: 

When we administered the first draft of the survey for the community profile, many people 
found it to be very difficult-too long, too wordy, not applicable if the person was not from a 
service provider agency. I approached (the project leader) reluctantly, telling her that there 
were some problems. She said, "No problem, tell me about it. That's your role; you are a 
pilot site.* She was very good about using the feedback, and also very good at providing 
feedhack to our memhers on the analysis from the self-study profile. She did it without 
making people feel bad. It was very constructive. 

Similarly, a staff memher at one of the pilot sites for the Mentoring Guidehook said: "The process 
was interactive from the beginning.... [Laboratory staff members) had great conferencing skills and 
really knew how to ask the right kind of open-ended questions. In other words, they modeled the 
mentoring process for us." A principal at a pilot site for Formative Teacher Evaluation said: 

Mast | outside consultants] want you to accept their program, hut [the laboratory staff] allowed 
us so much freedom ... hut we would still get help and support at a critical time.... They 
respect practitioners' opinions and views. That attitude is not always present at the university 
Icvel-they'll tolerate you. 

A staff developer who has led case discussions told us, Tve often heard |a lahoratory staff memher] 
say how much she has learned from us"; this participant also called the lahoratory "a great resource." 
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As these quotations suggest, the mutually respectful interactions hetween participants and 
laboratory staff contributed to an interesting phenomenon: by involving field sites in the development 
process, laboratories can very effectively deliver technical assistance to those sites. Each of the 
individuals quoted ahove (and many others whom we interviewed) made this connection more or less 
explicitly. The participant in Integrating Education and Human Services pointed to an exchange of 
feedback on an even footing, where site personnel gave the laboratory a critique of the draft 
materials, and then the laboratory constructively summarized the analysis of the site's profile. The 
participant in developing the Mentoring Guidebook said the laboratory staff behaved like skilled 
mentors. The principal who pilot tested Formative Teacher Evaluation praised the laboratory for 
simultaneously conveying respect and providing help and support. The staff developer involved with 
case discussions recognizes that the laboratory staff learn from the field at the same time its they are a 
resource. 

In these sites and others we visited, a relationship of reciprocal assistance was a somewhat 
surprising experience for participants, hut they spoke highly of such a relationship as the context for 
receiving assistance. In the Partner Schools, where a reciprocal relationship is a cornerstone of the 
activity's design, many participants responded positively to it. As one said: 

The lab has great ideas, but this project is strictly give and take. There's real interaction. 
The lab always wants to know what worked for you. They're drawing best practice ideas 
from us-this is not a one-way project. 

The relationship between the laboratory and the participants at implementing sites is important 
in two ways: it contributes to development; and it constitutes a major segment of the assistance thai 
the laboratory offers in its region. Thus, the way participants come to he involved in these activities 
is worth examining. The process varies in formality and in the criteria applied, explicitly or 
implicitly, to the selection (and self-selection) of participants. 

How participants are recruited. The outreach efforts that alert participants to the possibility 
of working with a laboratory can include conference presentations, workshops, or the preparation and 
dissemination of brochures. Laboratories also work with other agencies and organizations to recruit 
participants: professional associations help recruit and select participants in Study Groups; state 
departments of education have helped with recruitment for several of the activities, including Strategic 
Reading, the Mentoring Guidebook, and Partner Schools; in the case of Applied Research in 
Restructuring Schools, a state department of education initially required schools participating in the 
Carnegie Corporation's Middle Grade School State Policy Initiative to work with the laboratory. 
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In Creating the Future and Integrating Education and Human Services, most of the sites (or 
individuals in their leadership) had previously worked with the laboratory on one or more other 
projects. Individual working relationships have been part of the basis for participation in Case 
Methvxls; several key participants have worked with one of the laboratory staff members since before 
her employment with the laboratory. And one site traces its participation in Formative Teacher 
Evaluation to the fact that a teacher at the school was enrolled in a graduate course with a laboratory 
staff member, who announced to the class one evening that the laboratory was looking for participants 
in a new project. 

Sometimes laboratories formally select participating sites from a pool of volunteers. The pilot 
sites for the Mentoring Guidebook, initially' nominated by their state departments of education, had to 
submit applications. Similarly, sites applied for participation in Strategic Reading. In that case, a 
distinctive feature of the nomination process was that it included need as a criterion: the laboratory 
asked state departments of education to identify high-poverty, isolated sites; then the laboratory 
worked with the local superintendents to invite schools to submit proposals for participation. 

As participants looked back on the way they joined a laboratory effort, those who told us that 
the activity would fill a need tor them were outnumbered by those who pointed to their capacity to 
take on the role of participant. Many explained that they were already embarked on the kind of effort 
the laboratory contemplated. A principal who participated in QUILT described the confluence of 
need, capacity, and philosophical agreement this way: "I think what convinced the teachers is the 
emphasis on thinking skills. They were troubled that the kids are not taking time to reason. Also, 
this is an innovative faculty and they saw the potential for professional growth." Another typical 
comment about shared philosophy is that of a principal of one of the Partner Schools, who said the 
laboratory's principles for the effort meshed well with those the school had already developed: "It 
was compatible with the way we were thinking, a good fit." 

Still, participants varied in the extent to which they initially understood what they would be 
doing with the laboratory. In some cases, such as Creating the Future, laboratory staff made a 
concerted effort to acquaint would-he participants with the plans and procedures for the activity. A 
laboratory staff member said: "We want them to know what they are getting into, what we will do 
for them, and what they have to do. To the extent possible, we don't want there to be any 
surprises." 

A different experience was reported by some participants in the Applied Research on School 
Restructuring. Perhaps because the laboratory initially started working with them at the behest of the 
state department of education, several of them saw the laboratory staff as monitors for the state. Only 
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over time did they come to understand the purposes and scope of the intended activity, and some 
eventually realized they did not have the same expectations for the working relationship as the 
laboratory staff did. A principal said: 

1 was concerned that some of the information from our project ended up in an article.... I 
didn't understand from the beginning, I felt they were |here to provide) a feedhack process 
for us, not to use the information tor their own research. 

This participant did find value in the laboratory's work* however, as did other participants. A teacher 
at another school commented about laboratory staff, They were useful, helpful. They were one of 
the better things we've had." The principal of still another school described the laboratory as a 
resource that contributed just what the school needed: 

They rarely present information that doesn't require people to think.... Their written material 
is great stuff.... Their early report provided really accurate insights at a time when we were 
making some major changes. 

The conflicting assessments suggest that, after a confusing start, the laboratory was ahle to develop a 
shared understanding of purposes in some schools but not others. 

Chances In Behavior Ta k e Place 

Unlike some other approaches to the dissemination of research findings -where awareness may 
he the most realistic aim-development is one that can reasonably be expected to lead to hehavioral 
changes in the field. The products and processes developed are geared toward instilling new 
capacities, with the built-in opportunities for trial and reflection that are associated with actual change 
in practice. We found reports of behavioral change, in some cases substantial in scope, that persists 
beyond the immediate laboratory intervention at many sites, (However, we cannot say how 
widespread such changes are across all sites touched by laboratories' development work.) We also 
cite here some findings of the laboratories' own evaluations, laheled as such. 

Participants do new things. Changes in classroom teaching are attributed to both QUILT and 
Strategic Reading, according to the participants we visited and to more hroadly based evaluations 
conducted hy the laboratories. Participants in QUILT show significant differences in their questioning 
behavior, compared with nonparticipants. The questioning techniques imparted hy QUILT training 
are wait time immediately after asking a question, wait time after a student's initial answer, questions 
at all intellectual levels, and redirecting questions to involve as many students as possible. Evaluation 
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has shown that the full-scale QUILT model (with collegiums and peer observations during the year) is 
successful in bringing ahoui change in these behaviors, and this is the model that the laboratory is 
now disseminating. The teachers and principals whom we interviewed echo the wider evaluation 
findings: how teachers ask questions in the classroom has changed because of QUILT, they say. 
Some teachers also commented that the process of developing "pivotal" questions improves their 
lesson planning. 

The pattern of change found in Strategic Reading is similar. The laboratory's evaluation 
shows that participants significantly outscored a comparison group in their use of "real questions" and 
of "strategy comments regarding reading/' The teachers we interviewed in Detroit and rural Iowa 
say that they use the Strategic Reading techniques in their classrooms, and they give many concrete 
illustrations that lend credibility to these self-reports. They also say what they are now doing less of. 
such as using worksheets. 

Other lahoratory activities have also heen associated with new hehaviors at the implementing 
sites. Teachers participating in the Study Group on integrated instruction are using this method, and 
some of them are adapting and using each others* lesson plans. PRESS participants, according to 
trainers, have taken a "hig step" toward increased interaction among teachers and other staff within 
schools. Some participants in Partner Schools spoke of specific changes, including the following 
reports: "Classroom interactions changed to a more questioning and positive climate. Teachers held 
more discussions with students and gave less negative feedback." "Assessment measures and report 
cards were re-evaluated grade hy grade." 

Teachers participating in Case Methods have engaged in new hehaviors, according to 
lahoratory evaluations and our interview data. In an evaluation of a seminar focusing on cases in 
multicultural education, some participants reported changing how they interact with students and 
parents: They listen more to students* concerns, spend more time communicating with individual 
students and parents, and interact differently during instruction. There is also some evidence of a 
change in the ways that teachers plan lessons" (Shulman, Sather, & Mesa-Bains, 1992). In studies of 
the teachers participating in case discussions in mathematics, lahoratory staff have found self-reported 
changes of the following types: trying out activities featured in cases; spending more time on difficult 
topics; and using a wider array of materials (Barnett & Tyson, 1993). An ongoing study that 
analyzes videotaped lessons from the classes of first-year and continuing case participants is revealing 
different changes in different classrooms. Examples include extending the range of activities availahle 
to lower-ability students, using manipulatives more extensively and in different ways, and giving 
students more opportunities to discuss their own and each others* explanations. 
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in our interview with teachers who have participated in and facilitated case discussions, some 
reported that they use their facilitation skills in leading classroom discussions; some also cited their 
use of other new techniques in mathematics teaching. They also reported a change in teachers' 
conversations with one another. One said that participating in case discussions has "changed the 
discussion in the teachers* lounge. The discussions would continue during the day." Another chimed 
in: " You've huilt that collegial feeling." And still another cited a structural change as a result: 

Middle school teachers who had been on the same staff for more than a year were discussing 
a case and realizing they were having the same problems with assessment They had never 
discussed these things before. It got to a change in how the math department would work. 

In districts that have adopted Formative Teacher Evaluation, laboratory staff estimate that 
ahout two-thirds of teachers participate in the system. (In an elementary school that we visited, tbe 
rate is 98 percent.) This means that they are choosing skills to work on and are using feedback from 
administrators, peer observation, or videotape as a means of developing these skills. 

A pilot site for the Mentoring Guidebook credited the pilot experience with stimulating 
changes in its preservice and inservice mentoring: the preservice program placed more emphasis on 
understanding adult development; the inservice program added new activities using data collection as 
a strategy in mentoring. At another site* the reported changes were more general; a typical 
description of the changes there was, "We are better staff developers." 

Goal setting and planning were affected at the sites participating in Creating the Future (which 
emphasizes the formulation of missions and goals). Participants point to the existence of new mission 
statements and student learning goals. In addition, a laboratory staff member observes that the goals 
are relatively sophisticated ones and that the leadership teams have shown skill in organizing a 
community dialogue to reach consensus about goals. 

Similarly, two of the schools that have been sites for Applied Research on School 
Restructuring have used data from the student survey in writing their school improvement plans. And 
the data use that results from :his activity is not confined to formal occasions such as the preparation 
of plans-at least in one school, where the vice principal reported: "The school is becoming more 
data aware. Teachers will ask, 'What did the kids think of this?" 1 

Community involvement with schools has increased at some PRESS sites, according to the 
PRESS trainers. They point to the significance of this change, explaining that the communities "used 
U) he negative; now they're much inore involved. They used to think of the school as a 'government 
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school'-that the Department of Education would take care of it. They are understanding their new 
role/' 

Several sites were already welt launched. Attributing change to the laboratory's intervention 
is not always straightforward, even in these sites where the products or processes being developed 
were often quite concrete and where the involvement with the laboratory was typically intense and 
sustained. An important reason is that several of these sites had already embarked on the kinds of 
changes that the laboratory wanted to bring about.* For example, pilot sites for the Mentoring 
Guidebook were already actively committed to mentoring programs. As a respondent explained it to 
us: "You need to understand that we saw this as a way of beefing up and re invigorating our 
program, not starting something new. New ideas, yes; reshaping program, no." Partner Schools 
were selected for participation on the basis of their readiness tor change. Similarly, the sites for 
Integrating Education and Human Services were already forming collaborative relationships among 
agencies. In each case, experience and capacity equipped the sites to contribute to development. For 
our analytic purposes, however, the independent contribution of the lahoratory*s products and 
processes at these sites becomes hard to trace. 

Broader Effects Also Result 

Along with the intended changes in behavior at the sites using laboratory products and 
processes, other kinds of effects also ensue. Of particular policy interest are the changes observed in 
student performance, and we describe these here. We also discuss the additional effects described by 
our interviewees, which include broader individual and organizational learning and new interactions 
with networks of practitioners. 

Student effects. The most compelling evidence of the effectiveness of a laboratory's work 
would be demonstrable effects on student performance. In this group of development efforts, for a 
variety of reasons, such effects are hard to find. Some of the activities are too new; they are still 
undergoing initial development and cannot appropriately be held to the standard of demonstrating 
effects on student behavior. In this category are Partner Schools and Integrating Education and 
Human Services. Some are intended to intervene at a level considerably removed from the 
classroom; although effects on student performance could reasonably be expected at some point, they 
would take time to emerge and would be very hard to trace to the laboratory's product or process 
because so many other variables would inevitably intervene. Creating the Future and PRESS are in 
this category; Integrating Education and Human Services will he when it is fully developed. 
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This leaves several activities for which student effects could reasonably he traced. The direct 
participants in these activities are teachers, for the most part, and behavioral effects on teachers are 
expected (and often documented), k Within this group, those that have most actively looked for student 
effects are the two that are aimed at bringing ahout specified effects on teacher behavior, QUILT and 
Strategic Reading. The other activities instead encourage each participant or group of participants to 
pursue a different agenda for improving their own practice (Study Groups, Case Methods, mentoring, 
applied research in restructuring schools, and Formative Teacher Evaluation)--and these activities 
have included less attention to effects on students. 

In QUILT and Strategic Reading, there is a conceptual model linking new teacher behaviors 
to new student behaviors and then to improved performance.. In the other activities, such a model has 
not been designed. This does not actually mean that improved student performance could not be 
sought and assessed; the participants in case discussions in mathematics, for example, might each 
choose different teaching skills to develop-just as the participants in Formative Teacher Evaluation 
usually do-but they would presumably share the ultimate aim of positive effects on student 
performance. Nevertheless, only QUILT and Strategic Reading have done much to pursue evidence 
of improved student performance, and one reason may be the tighter specification inherent in their 
models of change. 

An early evaluation of Strategic Reading compared the reading perfon uice of students in 
districts where all third-grade teachers participated in the program, where some third-grade teachers 
participated, and where no teachers participated. Students in the first two groups significantly 
outscored those in the third group on one subscale of the Wisconsin Third-Grade Reading Test-the 
one that measures "reading strategies." They did not differ from the comparison group on other 
reading sub scales (comprehension, prior knowledge, or interest). Subsequently, the laboratory 
administered assessments to 220 students in seven urban classrooms that included some using 
Strategic Reading and some not using it. The available results focus on the responses of students and 
teachers to the tasks included in the assessment: for example, non-Strategic Reading students "had a 
hard time responding to 'thinking* type problems. They repeatedly asked for the 'right way to do it,' 
the one correct answer." By contrast, Strategic Reading students said they liked these problems and, 
instead, expressed discomfort with the restrictions in other types of tasks (e.g., listing five things or 
using a ready-made web in activating prior knowledge). 

In our data collection, a principal cited improved student scores on the Iowa Test of Basic 
Skills and attributed the improvement to the Strategic Reading strategies. Another pointed to 
improvements in grades: "Sixteen of 24 students whose teachers went through the training are on the 
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honor roll this year, and the other students had the grades hut had had citizenship. This | academic 
performance! was due to the strategies." 

The laboratory's evaluation of QUILT shows that students of participating teachers, compared 
with students whose teachers received only a single inservice session on questioning, respond with 
evidence of higher cognitive performance in videotaped classroom discussions. In our site visits, 
participants in QUILT expressed surprise that effects on student performance have not emerged in 
standardized assessments to date. The two principals whose schools we visited said that their 
students' writing has improved noticeably. They regret that the standardized test now in use does not 
include writing, and they look forward to promised new assessments that will allow their students to 
demonstrate this skill. 

In Case Methods, although the laboratory has not systematically sought measures of improved 
student performance, a teacher participating in one discussion group offered such evidence, which the 
laboratory included in a report. This participant in a group focusing on cases in student diversity 
cited an increase in her students' math scores. She attributed the increase to changes in the cues she 
gave students-a shift from "nebulous, middle -class goals* to short-term goals and the use of more 
specific instructions and concrete record-keeping systems. The student whose scores rose most 
dramatically was one who himself commented: "You said we were going to keep a notebook, but I 
really didn't know what a notebook was." The teacher continued: "After working with him and his 
muthe r , he knew what a notebook was and kept a great one, and I think the results showed up in his 
test scores" (Shulman et al., 1992). 

In the sites we visited, teacher reports of changes in their students' skills and behavior 
abounded. Teachers participating in QUILT gave the following reports: 

Students in second grade speak more in hill sentences because I give them the time they need 
to think through an answer. 

Fourth graders talk more with each other. They are aware of the concept of wait time and 
use it in their own discussions. 

Children are becoming better writers because they organize their thoughts. Their journal 
writing shows evidence that they are becoming better thinkers. 

I'm getting participation from students who never gave answers before. 
Teachers participating in Strategic Reading offered these observations of student skills and behaviors* 
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Kids are writing better. They are moving up to |heing| competent writers. I see a big 
difference in portfolio materials from the beginning of the year to the end, especially in the 
organization because of the focus in SRP. Kids are thinking through what they want to say. 

One hig thing is that kids are thinking ahead more. There is a lot more conversation in 
classes among the students and between the students and the teachers. 

Students are coming up with interesting questions and a purpose for learning. They are 
starting (o want to read to learn. They are connecting reading and science. 

Kids are more open to give answers whether they're right or wrong. They are catching on 
better, reading more, going to the library more to get hooks, not just to play. 

Two teachers participating in Formative Teacher Evaluation gave an unconventional and 
intriguing example of new student behavior. In their enthusiasm for feedhack about their own 
teaching performance, they enlisted their students as observers and critics of the specific behaviors 
they were trying to develop. Even the kindergartners could help their teacher monitor her hehavior 
and offered comments about her teaching at the end of a unit. She reported: It helps the children 
see we're lifelong learners. I tell them I'm working so I can be a better teacher." A sixth-grade 
teacher also used this technique, and she commented: "They'll see themselves and me [on the 
videotape] and point out things. They're helping me, teaching me.... They see you improve and 
they see that what they had to say was important. " 

Broader individual and organizational learning. Some development efforts have been aimed 
at fostering a broad disposition to inquiry and learning among participants. Others have not had this 
aim at the forefront of their formal purposes but have nonetheless achieved it: an intervention that 
starts with a particular, concrete, set of behaviors to be imparted can trigger a hroader disposition to 
inquiry that enables individuals and organizations to learn more than what they set out to learn. 
Examples can he found in several of these development efforts. 

The Partner Schools have participated in an R&D activity with very broadly defined purposes, 
and those we visited have, for the most part, responded with a sense of broadened personal and 
organizational horizons. A participant said: 

We are looking at schools differently. We are looking at children as learners, adults and 
community as learner. We are a learning community. The lab experience has given us time 
to address some issues and make some changes—to full inclusion, new ways of reporting to 
parents without report cards. They helped us map our journey. 
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Comments from the sites that pilot tested the Mentnring Guidebook strike a similar chord. In 
one site, in particular, the principles of mentoring are said to have revitalized some teachers; student 
teachers are said to have become more "self-reflective" and more interested in "discovery." This 
activity is also notahle for the depth of its reported effects or -irticipants in the working group that 
developed the initial version of the guidebook-state education agency staff members with 
responsibilities in professional development. Participants credited the experience with greatly 
increasing their knowledge base and helping them better understand what other states are doing. One 
observed: 'This Ls possibly one of the best professional development activities I've ever been 
involved with, because it gave me the opportunity to be in a Study Group, to write and reflect. ' 
Participants look hack proudly at the ambition of their undertaking: "When the group formed, people 
didn't realize how big a project it would he and how sophisticated they would become/ 

Some participants in Study Groups, Case Methods, and Formative Teacher Evaluation cited 
changes in their overall orientation to their work. A participant in a Study Group on interdisciplinary 
instruction repoits a growing understanding thai this kind of teaching requires teachers to do a lot: 

They have to keep up with new content, the related political perspectives and events, as well 
as current pedagogical philosophy and instructional techniques.... Teachers must reach a 
higher level of understanding of their discipline and its connections to other disciplines and 
current events; they become learners. 

A participant in case discussions, quoted in a laboratory report, said: "More than ever before I sec 
teaching as ongoing problem solving. Yon have to continue to work on your instructional strategies 
to fit the kids you have. And you have to keep what works and get rid of what doesn't" (Shulman et 
al., 1992, p. 15). A teacher whom we interviewed observed: "The [case] discussions made me perk 
up and look for other training things. I started to be a math person." In an elementary school where 
almost every teacher participates in Formative Teacher Evaluation, we were told: "The school is 
working as a unit. It brings a large campus together." 

Although QUILT looks more focused than these other interventions because it is intended to 
impart a particular repertoire of teaching skills, the positive observations of participants about their 
learning were not different in kind from those we found in studying the broader-purpose 
interventions. Indeed, a principal whose school has participated in QUILT said: 

This is a perfect, complete process of teach-model-model back. It's a total professional staff 
development program, not just about questioning, but about what teachers do all day. There 
is no college course on how to be a good teacher-the process part. Education is fragmented. 
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QUILT is the process model. It builds teachers as individuals and strengthens the sense of 
community within the school. Calling it a program about questioning is selling it short. 

Networking. The opportunity to interact with peers from other sites was often mentioned as a 
beneficial result of working with a laboratory. Some activities, such as Study Groups, incorporate 
this as a key feature of their design. Participants in Study Groups cited the exchange of ideas as a 
benefit. 

Two other activities that have been especially effective in opening lines of communication 
among participants are the Partner Schools and the Mentoring Guidebook. Through formal 
mechanisms such as the steering committee and thematic working parties, and through the 
laboratory's informal encouragement for cross-school communication. Partner Schools participants 
have had many opportunities to build relationships with colleagues and to use them as resources. 
This represents a profound change in standard operating procedures for many schools. For example, 
the principal of a small rural school said: "I never would have thought of calling [a community some 
SO miles away] before this. Now 1 know the people there and I pick up the phone in a minute. I 
really believe that the networking will long outlast the project." Another principal reported that the 
school is different from the rest of the schools in its own district because of its innovativeness, and 
that as a result, "we liked the opportunity to network ... we no longer feel as isolated." 

Participants in the development and testing of the Mentoring Guidebook are especially 
noteworthy in their enthusiasm for this activity's communication and networking opportunities. 
Members of the Teacher Induction Working Group (the state- agency staff who wrote the guidebook) 
welcomed the chance to share information and ideas across stais boundaries and to forge relationships 
with their colleagues in other states. They credited the laboratory with setting the tone for a collegia! 
atmosphere of mutual respect, rather than any hierarchical structure, and with letting the group "set 
our own course and direction. 11 Respondents spoke of the importance of sharing information and 
discussing ideas witb other states. Opportunities such as this project that involve lengthy 
conversations among states are unfortunately very rare, they said. The relationships have continued: 
as one member commented, "I fee! very comfortable about calling these states now about other 
issues." The pilot sites for the guidebook, whose representatives met periodically at the laboratory, 
also spoke highly of the networking opportunities. Indeed, the working group and pilot group have 
agreed that the logical next stage for this activity is the formation of a Regional Mentoring Network 
that will extend the benefits of such meetings to a wider group of participants. 
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Limitations 

Development cannot do everything, and we have some findings ahout what it did not do in 
these instances. Two areas of concern emerge from our data. First, the critical importance of 
sustained, in -person technical assistance during implementation helies any expectation on the part of 
policymakers that development may offer a low-cast route to education reform. Unlike a simple 
consumer product, a product or process developed by a laboratory cannot be widely disseminated for 
do-it-yourself installation. Second, those development efforts that are aimed at effecting behavioral 
change in a broader sphere than the individual classroom (or a group of classrooms in a school) seem 
to pose particularly acute challenges. Based on the data we have collected, such efforts seem prone to 
be either too narrowly prescriptive or too amorphous to yield entirely satisfactory results 

The Expense and Uncertainty of Going to Scale 

As a development activity matures, the issue of its future design and utility becomes more 
salient. Some of the activities we studied are still at earlier stages of evolution (most notably Partner 
Schools and Integrating Education and Human Services), but the others are confronting in different 
ways the question of how, if at all, they can "go to scale." This phrase is acquiring mystical 
properties in the current policy conversation, but in the context of a laboratory development project 
we understand it to mean the widespread implementation of the products and processes developed. 

We can begin this analysis with a straw figure: the assumption that a development project 
will generally produce some sort of do-it-yourself kit for educators to implement in multiple sites with 
minimal human intervention from the developers. Our findings indicate that this is almost certainly a 
had way for policymakers to think about laboratory-developed products and processes going to scale. 

Furthermore, only one activity in our sample, QUILT, has as yet succeeded with a 
dissemination strategy in which training is handled by trainers and the laboratory does not have to 
have a continuing technical-assistance role. In QUILT, the central outcome of implementation is 
described in specific, behavioral terms (i.e., teachers use particular questioning techniques), and the 
training designed to bring ahout this outcome has been thoroughly scripted as well By applying a 
high level of engineering to a treatment that is designed to produce a well-specified outcome, 
QUILT's developers have demonstrated success in handing off the activity for second 'generation 
dissemination through a network of trained trainers. In our view, however, this accomplishment 
rni^ht be hard to replicate with products and processes less connected to the hehaviorisi tradition. To 
the extent that a laboratory develops products and processes that accommodate more local variation 
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and reinvention-packages that are intended to he unpackaged-a dissemination process cannot he 
engineered to cover every local contingency. 

More commonly in this sample of activities, the laboratories have a general intention of 
bringing about widespread use of the products and processes they develop, and they seek to enlist 
committed participants in this effort. However, we have found mixed results in this second- 
generation dissemination, with local context and individual commitment having a powerful effect. In 
Strategic Reading, for example, leadership teams are expected to impart the program's techniques to 
their colleagues. Some do; some do not. Among the schools we visited, building-level commitment 
to the aims and methods of Strategic Reading (especially on the part of the principal) makes a key 
difference. Persistence can also help to overcome initial skepticism, as illustrated by the comments of 
a teacher in an urban middle school: 

We give teachers assignments and they do them. The teachers are psyched. This year's 
group is more cooperative than last year. Last year they were nervous the program would go 
away, but now they have the testimony of last year's group. 

Variation also appears within the Partner Schools. (Although the laboratory's overall 
development effort is many years from completion, participants have been encouraged to form design 
| teams in their schools and to share specific ideas gleaned from Laboratory-sponsored events,) 

Participants have found varying degrees of receptivity to their new ideas within and beyond their 
schools. 

When the strategy for dissemination is informal and relies on volunteers, its power is likely to 
| be limited. Teachers and administrators participating in Formative Teacher Evaluation are enlisted in 

dissemination, but out interviews suggest that their volunteer efforts can be stretched only so far. 
| Teachers grumbled about the work involved in making presentations in other sites, and we suspect 

' that their willing participation in these events will diminish soon. 

| The use of Case Methods is beginning to diffuse widely, through a variety of channels, and 

the laboratory Ls working out a balance hetween closely monitoring its expansion and allowing it to 

| snowball. We interviewed teachers who have participated in the activity for some years and are now 

facilitating discussion groups. As experienced teacher leaders who have joined laboratory staff in 

(writing articles or making conference presentations about Case Methods, these teachers could be 
characterized as members of a virtual project team. The laboratory is now grappling with the 
tradeoffs between keeping close tabs on the growth of Case Methods, especially in mathematics, and 
| allowing wider dissemination by individuals less closely affiliated with the developers. 
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Specific disappointments have emerged in the implementation of two specific plans for 
dissemination. Strategic Reading originally had a plan of using technology to create networks of rural 
teachers for continuing professional development. For a variety, of reasons (lack of hardware, 
scheduling difficulties), on-line networking has not taken off. Products can also disappoint. The 
Mentoring Guidehook was intended to be a stand-alone product, hut those involved in its development 
and piloting have concluded that an 800-page notebook of research summaries and activities does not 
effectively stand alone. As a result, the process (a network of interested sites) has taken precedence 
over the product. A participating teacher analyzes the strengths and weaknesses of the overall effort 
this way: 

Have you seen the finished product? We kept saying it was too long. It's still too long. The 
training was great, the whole experience was great, hut if 1 got that hook in the mail. I'd say. 
"Jeez, what is this thing?" 

Whether these two examples illustrate problems of project design or execution is hard to say. 
For whatever reason, an intended dissemination channel doesn't work out as planned, and the fallback 
plan is a continuing, hands-on role for laboratory staff. Indeed, hands-on assistance from the 
laboratory is part of the future plans for virtually all of these development efforts. Only in the case 
of QUiLT has second-generation dissemination really lived up to expectations, in other cases, the 
laboratory expects others to assist in dissemination and assistance but also expects to have a 
continuing role itself. Thus, policymakers should think of development as part cf a complex of 
interrelated activities, with continued technical assistance very likely to be part of the mix. 

The Pitfalls of Development Aimed at Organizations 

Although success stories can he found among the examples of development efforts that are 
aimed primarily at affecting classroom behavior, current federal education policy has a different 
focus: it takes the system as its unit of intervention. Unfortunately, we can say little about the likely 
contributions of development in this arena. The development efforts aimed at schools, districts, or 
wider systems are relatively new. and their payoff cannot yet he assessed. All we can do is to 
identify the types of problems that these projects are now striving to overcome. They may prove 
successful in doing so, but in any case an explicit identification of the pitfalls may he helpful for 
future R&D policy and management. 

Highly concrete projects may be too concrete to connect to wider issues in implementing 
sites. Creating the Future illustrates this pitfall in development efforts that are designed to affect 
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district-level hehavior. The tangihle character of its materials and activities is hoth a strength and a 
weakness, according to participants. There are times when they welcome the structure ottered hy 
Creating the Future materials and standard operating procedures, and they acknowledge that these 
reflect a sensihle distillation of research. The leadership learns use the training to carry out a set of 
activities in developing mission and goals that can hring order to the putentially disorderly process of 
drawing out a district's and community's ideas. Yet several people we interviewed expressed concern 
ahout the amount of formjlaic prescription hrought to hear on these tasks. The following comments, 
from a principal and a superintendent, are typical of the criticisms: 

Onward to Excellence and Creating the Future are good resources in terms of process and 
research and theory. Some of the training was good, hut I wasn't that impressed hy lhe early 
sessions. The lah relies too much on pre-packaged materials and processes. They need to 
develop a more experimental approach. 

There was a sense that people would have liked more guidance, hut not more overheads. 
When something pops up in the environment, (the lahoratory staff] can't help. They can't 
answer the questions ahout what has worked in other communities that have tried these things. 

We were also told about the limited usefulness of Creating the Future at the next stage, where 
the district must assess the extent to which current programs are helping students achieve the learning 
goals: "After you get past the essential learning goals, there is not as much help. The first three 
steps were easy. After the first three steps, I don't get a message." Two circumstances may account 
for this perception among participants. First, the field of assessment is in turmoil now; a district that 
rejects standardized tests as too limited and confining has few proven alternatives to rely on. The 
second reason is inherent in Creating the Future: the process of setting a mission and goals is, to 
some extent, a discrete project that unfolds in isolation from the district's ongoing work in teaching, 
learning, and assessment. Thus, when the time comes to tackle the core functions of the 
organization, no strong connection to these functions exists. 

These dissatisfactions with Creating the Future illustrate the difficulty of developing a training 
package that can usefully equip a district to identify and address issues in a changing environment and 
with direct implications for the work taking place in every classroom. Perhaps inevitahly, the 
standardization of the package brings two unwelcome side effects-a lack of flexibility "when 
something pops up in the environment/ and a tendency to emphasize process at the expense of 
s: instance. 

More amorphous development efforts can leave participants feeling they have received little 
benefit. At the other end of the spectrum are development efforts that eschew prescription. The 
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danger is that participants may view these activities as puzzling ventures that bring frustration and that 
border on insignificance. 

Comments from some participants in Partner Schools illustrate this prohiem. In this case, no 
one could accuse the developers of coming up with an overly packaged process that fails to connect 
with its local environment. Instead, hecause the laboratory staff have emphasized continual flexibility 
in the change process, some local participants have been continually confused, as shown in the 
following comment about one school's design team: 

In the minutes of every single meeting you see at least one reference to the confusions we all 
had. What's our job? What's our mission? What are we supposed to do? The lab was like 
this giant psychiatrist. When you asked them what to do, they'd say, "What do you want to 
do?" The problem is we didn't know. 

Moreover, the ahsence of priorities, boundaries, or focus resulted in exhaustion for some participants, 
as a teacher in the same school described it: 

It's hard work always wondering if you're on the right track and working to change things 
without knowing if you're headed in the right direction. The lab never said to slow down, 
either. If we said we were thinking about doing something, the response was, "What are you 
waiting for? Go ahead and do it." So we wore ourselves out. 

Although the participant assessments reported here are among the most negative we found in 
this evaluation, neither the Partner Schools activity nor Creating the Future should be judged a failure 
on the basis of these remarks. Participants in each activity point to benefits they have derived, and 
laboratory staff are working to address the flaws they have found in each of these developmental 

efforts. 

However, the tensions of too much or too little structure are likely to persist in these and 
other efforts to apply the tools of development to organizational issues. A school district (or a school 
or a community) has a lot of moving parts, and developers who are trying to work at a genera) level 
run the risk of either denying the complexity or drowning in it. Unfortunately, the evidence we have 
been able to gather tells us little about effective ways of resolving this diiemma. 
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What It Takes 



Development is an ambitious business. The changes observed at implementing sites-new 
behaviors that persist in classroom leaching or organizational operations, some changes in student 
performance, and broader changes in individual or organizational dispositions toward learning-have 
resulted from concerted, large-scale efforts to apply and refine the educational knowledge base. 
These laboratory activities offer a number of lessons in the techniques of development, and we believe 
there is still more room for learning. 

Time 

All the activities reviewed here have heen of several years' duration. Most have spanned 
more than one contract period, in fact. Time is needed for products and processes to evolve through 
several development cycles, in which they can improve through the thoughtftil application of lessons 
learned in the field. 

To allow several years for development would represent a change in policy for the laboratory 
program. The 1990 RFP specified that development efforts should be designed so as to have results 
in the field within the five-year contract period* Although expecting field-test results would certainly 
be reasonable within a single contract period, we do not think that the government should expect 
development to be completed, widespread implementation underway, and results of that 
implementation observahle within five years. The gestation period for major products and processes 
is too long for this to be a realistic timetable. 

Although we advocate allowing a longer period for development, we would not encourage the 
laboratories to move slowly in development. On the contrary, we believe that the products and 
processes that laboratories develop would benefit from several rapid cycles of trial implementation, in 
close consultation with participants. Formative Teacher Evaluation moved into the field quickly and 
was extensively shaped by the trial participants during a single school year; we believe this pace sets 
a good example for other development efforts. (More complex products and processes would 
necessarily move through development mote slowly, but acceleration should still he the goal.) By 
contrast, some of the activities we studied have evolved quite slowly, sometimes because their broad 
array of purposes has posed difficult conceptual and practical tradeoffs (as in the Partner Schools 
activity), and sometimes because the laboratory staff and other participants have been busy with other 
things (as in PRESS and other activities). Especially when a development activity seems to be off to 

48 



BEST COPY AVAILABLE 



a promising start, we would encourage laboratory managers to invest enough staff resources to keep 
the activity moving. 

Engagement with the Field 

The evidence reviewed here suggests two points about engagement with the field in 
development activities. First, as we have emphasized throughout this chapter, the existing 
development work of laboratories is closely coupled to technical assistance. Laboratory staff provide 
a great deal of assistance to participating sites, and policymakers should not arbitrarily separate 
development from assistance in the design of the program's scope of work or in the performance 
indicators that laboratories are asked to report. Moreover, the nature of the assistance relationship as 
reported by participants-in which the exchange of ideas and opinions takes place on an equal and 
mutually respectful basis-offers a very interesting model that might be more widely applied in federal 
technical-assistance programs. 

A second implication of bur evidence is that laboratories should consider what sites can 
contribute to development, and then design their activities and select participants accordingly. The 
role of participants in development can range from that of experimental subjects in the classic 
tradition (i.e., to experience a treatment and then manifest behavior that the experimenters will 
measure) through pracJfrioner critics (i.e., to tell the laboratory when its materials are too long or 
M research-y M ) to full partners in inquiry (i.e., to take the laboratory's initial idea and actively adapt it 
while informing the laboratory about the nature and results of the adaptation). We believe the last 
role is especially promising and deserves further exploration. 

Indeed, the engagement of practitioners in laboratory inquiry fits a perspective emerging in 
innovative corporations, where R&D and marketing are being linked in new ways. This perspective 
is especially evident in the development of products that represent breakthroughs-which, we would 
argue, is the type of development that federal funding would most appropriately support in education. 
Rather than just allowing an R&D department to engineer new variations on existing product lines, or 
just surveying customers about their needs, some corporations engage customers more interactively in 
the design and refinement of new products-and, significantly, place their own technical experts in 
situations where they will learn about the daily realities that customers face. The director of the 
Xerox Palo Alto Research Center (PARC) writes: 

Most market research assumes either that a particular product already exists or that customers 
already know what they need. At PARC, we are focusing on systems that do not yet exist 
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and (in needs that are not yet clearly defined. We want to help customers become aware of 
their latent needs, then customize systems to meet them. Put another way, we are trying to 
prototype a need or use before we prototype a system. (Brown, 199 1, p. 1 10) 

Two researchers in business illustrate the importance of "needs that are not yet clearly defined" as a 
basis for forward-looking R&D with this analogy: *Go back a decade or two. How many of us were 
asking for microwave ovens, cellular telephones, compact disc players, home fax machines, or 
electronic whiteboards?" (Hamel & Prahalad, 1991, p. 85). They go on to describe several 
companies that are integrating in-depth marketing with R&D to create "a potent mixture of market 
and technical imagination" (p. 86). 

If laboratories deliberately arrange their development work so that they are learning from 
practitioners, they will need to choose their field sites appropriately, and this may raise some 
dilemmas. The self-selection evident among participants in many of these development activities 
typically rests on criteria of (1) philosophical affinity with the basic design (e.g., an interest in 
learning new questioning techniques so that the character of student participation in class will change) 
and (2) capacity to embark on a new venture. Need may enter in, but we find that participants rarely 
describe themselves-even in retrospeci-as very needy. Working with pioneering, reasonably high- 
capacity, volunteer sites has worked out well in the activities we have studied; it gives the laboratories 
insights that might not emerge from a more representative array of sites. This practice does raise two 
worries, however. First, products and processes that work well in very high-capacity sites may fall 
flat when disseminated to a wider array of sites. Second, because the assistance associated with 
development represents a large investment of laboratory resources, there may be a policy concern that 
more of these resources should be directed to more needy sites. Thus, we would suggest that 
laboratories continue and expand their efforts to find volunteers among high-poverty sites and to draw 
upon the capacities that exist in these sites. (Indeed, the fact that so many participants told us they 
had a high capacity for change may reflect the laboratories' unobtrusive assistance more than pre- 
existing, objective reality.) 

Evaluation of Field Trials 

The fact that field testing exists at all in the laboratory program is praiseworthy; it 
distinguishes the laboratories' work from commercial ventures of various kinds and educational 
innovations in particular, which are often cooked up in a vacuum, without benefit of a sound 
theoretical base or responsible reality testing. The laboratory program is committed to field testing 
and formative evaluation, and we urge that this commitment continue. 
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Across these development activities, we have found a range of practices in the design, 
analysis, and use of field testing. The evidence suggests to us that field testing should he based on 
relatively systematic designs; for example, the comparative test of varying designs embodied in the 
QUILT fie'd test offered a strong basis for the laboratory's further work with that program. Not 
every development effort will lend itself to a comparative trial of varying models, but laboratories 
could more consistently apply the general principle of setting up a field test to answer some specific 
questions, with documentation of the findings* Informal learning has a place in field testing as well, 
but it should not be the only process at work* Moreover, we believe the formal questions should 
include fundamental ones about the purposes and merits of the products and processes under 
development, as welt as questions about more marginal matters such as product formatting. 

We found some of the most thoughtful responses to field trials among laboratory staff 
members who analyze the field evidence through the lens of research and theory, as in the Case 
Methods activity. A theoretical framework helps make sense of the results found in the field and can 
guide the laboratory to appropriate revisions of the products and processes. In addition, laboratory 
staff who are contributing members of a research community can convey the findings of their field 
trials to others who need to learn from them. The fact that laboratories span the boundary between 
research and practice does not just equip them to communicate practitioners' needs to researchers; it 
also enables them to bring an important, applied perspective to theoretical discussions. 

Finally, we urge the laboratories to use the results of field testing. Although we found most 
developers to be conscientious about revising their prototype products and processes, human nature 
makes it hard to make major revisions or abandon unproductive lines of work, and we found few 
cases in which laboratory management has pressed hard for a rigorous, empirically based revision 
process. We think governing boards, executive directors, and other top managers should more firmly 
encourage the use of evaluation results and other field data. 

Investment 

Because development is expensive, an argument must be made for its cost-effectiveness. As 
we have seen, that argument cannot depend on a notion that the products and processes developed by 
laboratories will become self-winding devices for widespread, cheap implementation. The complexity 
of these products and processes requires elaborate provision for dissemination and technical 
assistance. For the most part, second-generation dissemination and assistance have not been very 
effective, although the scripted training of QUILT trainers is one exception and the gradual expansion 
of the Case Methods and Formative Teacher Evaluation families may prove to be others. 
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Instead, the argument for an investment in development must go back to a basic principle of a 
government investment in educational R&D: that advances in applied knowledge can spring up 
among isolated practitioners, but they will not amount to much until someone does the hard work of 
formalizing, testing, and refining them in multiple settings. At their best, laboratory development 
efforts of the kind analyzed here accomplish this purpose. They are undeniably costly, and we think 
that managers and policymakers should continually push for more efficient learning within and across 
projects, but an investment in development is a unique and defensible part of the array of federal 
policies intended to foster educational improvement. 
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III. SHORT-TERM INFORMATION EVENTS 
AND INFORMATION PRODUCTS 



Among the best known of the laboratory activities are those that require a comparatively 
limited commitment in time and resources for all participants. Because these interactions taken as a 
whole take up a relatively large proportion of effort at most laboratories, we decided that they were 
worthwhile targets of investigation. Broadly speaking, the activities we examined in this category 
comprise products and short-term events that impart specific, research-based information (for 
policymakers or educators) and promising practices (primarily for educators). Although all the 
activities discussed in this section fall under this rubric, the processes, products, and results can be 
further divided into two types, which we examine separately. 

Most laboratories make themselves available to policymakers in their region as an information 
resource and source of quick-turnaround technical assistance. This may include research syntheses 
tailored to particular requests, new papers, packets of published materials, or consultations. What 
these efforts share is the quality of quick responsiveness to the needs of key stakeholders. 

The second category includes products and short-term workshops, some of which represent 
distillations of far more ambitious activities such as those discussed in the previous chapter. 

Assistance to Policymakers 

All laboratories respond in some way to policymakers in their region; we know that when a 
governor's office calls to ask about systemic reform, the caller will certainly be referred to someone 
to help out. In probing further, we discovered that all of the laboratories are equipped and willing to 
give this kind of assistance fairly often, while at least one (Ed-Aide at Southwest Educational 
Development Laboratory [SEDL]) has carved out its own distinctive program to play this role. 

When we decided to gauge the laboratories' responsiveness to key officials in their regions, 
we originally looked for the Ed-Aide approach; for example, a legislative liaison calls to find out 
about cooperative learning and receives a customized packet of articles and research summaries that 
provide a quick map through the topic. We learned, however, that laboratories respond to inquiries 
in many ways, so we broadened our definition considerably to include action in response to an 
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information request, generally including a written product, which may or may not be assembled as a 
direct consequence of the request (i.e., may not always be "tailored"). 

We also discovered that there was a great deal of variation in the length of the interaction; it 
might consist of a few articles sent by overnight ma : J or documentation of the proceedings of an ad 
hoc committee developed to address the issue for state officials. These encounters are thus not 
entirely comparable in amount of effort or expected outcomes. The reader should also keep in mind 
that, as in the previous chapter, this category represents an analytic construct of our own and does not 
reflect any design developed either by OERI or by the laboratories. 

We interviewed about 40 state officials by telephone during the summer of 1994. Because we 
drew from lists provided by each laboratory, the responses can not he considered a representative 
sample of recipients of this kind of service. In some cases it was evident that laboratories had 
reminded clients of the interaction and in others that they had not; as a result, the informants varied 
in their degree of preparation for the interview. In only two cases were the people we interviewed 
unable to remember the exchange that the laboratory had described. 

We list below a range of requests from policymakers, along with very brief summaries of the 
laboratories' responses. All of these individuals call themselves more or less "satisfied customers/ 

The laboratories often answer inquiries by preparing a set of relevant, previously published 
materials, as the examples below illustrate: 

• A director of a state association of school administrators wanted information from 
AEL about the proportion of funding that goes to administration. She received a 
package of materials with funding statistics from the laboratory, which helped confirm 
the argument her agency was making to teachers and schools -that the claim that an 
outrageous amount of school funds goes to administration Is largely a media-created 
myth. 

• A state policy advisor needed to provide the governor with background materials on 
charter schools. FWL sent a package of related articles. 

• A state education agency (SEA) staff person needed quick assistance when she was 
suddenly put in charge of early childhood work and knew nothing about the topic. 
SEDL sent packages of articles and research that helped her become more comfortable 
in her new field. 

We also found a number of cases in which the laboratory wrote a new document, tailored to 
the recipients' needs: 
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• A former state superintendent asked NCREL for background information on 
technology in classrooms to present at a technology task force. The paper the 
laboratory provided helped the committee get started on its work; it was also used 
later for the governor's summit on technology and disseminated statewide. 

• A commissioner of education, for an effort to boost Native American achievement, 
needed to make a strong case to the state board on the importance of early language 
acquisition. NWREL wrote a report that summarized the research on the role of 
language development in schooling, which encouraged the board to push for 
appropriate legislation. According to the commissioner, this document turned out to 
be a seminal piece in the field. 

• A director of policy for an SEA needed a list of "demonstrably effective strategies" 
for schools seeking extra funding. RBS helped put together a document that described 
generic strategies along with the supporting research. According to this official, the 
report is now used by all 436 "special needs" schools in the state. 

• A deputy commissioner asked NE/IS for help in documenting the state's school 
restructuring initiative. This turned into a two-year collaborative effort resulting in a 
widely disseminated report. 

Sometimes policymakers' requests are focused enough to be answered by phone conversations 
other brief communications: 

• A legislative liaison who specializes in disability issues put nut * request for 
information on different states' funding formulas for special education. This inquiry 
was made over an electronic bulletin board. Since she was not in the field of 
education, she was surprised when Mid-continent Regional Educational Laboratory 
(McREL) responded with all of the information the staff knew to be available on the 
subject. She noted that there is no equivalent information resource in her field. 

• While revising teacher preparation policy, a director of a state legislative commission 
needed information on how other states fond their programs. SEDL sent thumbnail 
sketches of what was going on elsewhere and provided names to contact for further 
information. This overview confirmed what the state staff person had suspected: that 
few states were active in this area yet. 

Most common in our sample were responses that used a combination of the above strategies: 

• A director of high schools for a local school district wanted some information from 
AEL about block scheduling -specifical ly , approaches and strategies that had been 
tried successfully elsewhere. The laboratory sent a package of information and 
followed up with some on-site assistance. 
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• A state department of education official asked FWL for assistance in preparation for 
GOALS 2fl00. The laboratory sent materials and made two trips to work with the 
agency on the effort. 

• A director of professional development for an SEA, while working on recertification 
activities, asked NE/1S to document how recertification was linked with professional 
development in other states. The laboratory wrote a paper- in response and helped 
facilitate a policy seminar on the issue. 

• For a report to the governor, an SEA public information officer needed materials on 
the extent and results of school choice nationwide. NWREL prepared a document 
summarizing choice programs nationally along with related position papers. 

• A director of a state board of education asked RBS for help in revising the existing 
curriculum and testing regulations. Laboratory staff helped draft testimony to the 
state legislature and provided many other working documents. 

• A director of pupil services for an SEA needed assistance with improving student 
motivation and discipline, RBS helped put together a task force and contributed a 
range of written materials. 

• A research analyst for a state legislature asked SEDL for help in drafting legislation 
on a service learning program. The laboratory sent a combination of summary pieces 
on the topic and a copy of the relevant federal legislation. 

The examples listed above, from the perspective of the participants, represent success stories 
for the laboratories. From most of the people we interviewed, the only complaint we heard was that 
the laboratories are unable to put more time and effort into state support, since they are so well- 
equipped to do so. 

Inevitably, even in a group selected by the laboratories themselves, there were a few 
disgruntled users. One deputy commissioner who requested background information for an upcoming 
policy decision felt overwhelmed by the tons of paper that his request elicited- "too much for anyone 
to possibly read"— and none of it tailored to his state's particular situation. Three policymakers noted 
that the laboratories were great at providing information and assistance on areas that fit "their own 
agenda," rather than responding to me expressed needs of the states themselves. Finally, two SEA 
staff people complained that laboratory staff were very uneven in quality: some were able to respond 
sensitively while others were "damn abrasive." Indeed, many people we interviewed qualified their 
praise by saying that their terrific experiences during these encounters with the laboratory may have 
been personality-based rather than characteristic of the institution as a whole. 
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Results 



As a whole, the group was very pleased with laboratory responsiveness, and most respondents 
noted that they had put the information to immediate use (eg,, highlighted passages and sent them on 
to the governor; used as background for drafting legislation; confirmed what they suspected all along; 
etc,)- While we did not independently verify these reports, it seems reasonable to conclude that the 
laboratories generally make a successful effort to respond to state officials' policy inquiries in an 
effective way. While we found no officials who had been soundly rebuffed in their requests, many of 
our respondents were familiar with other laboratory connections in and around their workplaces, and 
none had dire news to report. 

The information is supplied promptly and is well suited to the need. Most of our 
respondents mentioned first how soon they received the answers to their requests (which were often 
about very time-sensitive issues). Only one official said that he or she was still waiting for the results 
of a request made long ago. More commonly, the individuals we interviewed echoed the response of 
a state legislative aide wbo said, "The materials arrived quickly and they were exactly what we asked 
for," 

The information is of high quality and presented in a useful format Most respondents 
highly praised the written materials and assistance they received. With the one exception described 
above, all noted that a great deal of complex information was usefully distilled to be reader-friendly 
for busy staff. Respondents mentioned that the work often seemed to be up to date and on the 
"cutting edge" compared with materials they were able to gather elsewhere. Possibly because the 
information is so well targeted to its audience, our group of policymakers typically praised seemingly 
contradictory qualities: the depth of the information and its readability. 

The information is free from bias. Several policymakers appreciated getting information 
that was not slanted toward any policy option, but rather that created the basis for an informed 
decision. As an SEA staff member put it, "They always give us both side of an issue and let us 
decide. Once we've chosen our own path, they will help us get there—but they are really more 
process-oriented." Moreover, this unbiased perspective adds value to the information itself: "Their 
neutrality lends credibility to our work" (another SEA staff person). Whether required to add 
ammunition to an already solid policy position or to structure future options, the laboratory seems to 
represent a highly trustworthy resource. 

The information may be a starting point for other hands-on assistance. In several of the 
events we heard about, the materials were followed up with phone calls, visits, creations of 
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committees, even large conferences. The interaction often provides an important linking mechanism 
for colleagues otherwise unknown to each other, and may help form new networks for people 
working on similar issues. Most respondents said at the very least thai they passed the information on 
to others in their organization, who then typically might meet on the topic, make a decision, and then 
ask for further assistance. Although sometimes meant only to provide a quick and easy solution, 
these customized encounters occasionally lead to iterative processes in which the laboratory's help 
persists and changes over time. A director of pupil services for an SEA described it this way: "They 
don't just drop their load and then leave/ 

The interaction serves important functions for the laboratory itsetf. Conversations with state 
policymakers form a significant pan of laboratories* ongoing, informal needs assessment for their 
regions. When information and laboratory-developed materials are passed around among colleagues 
or used ia task force meetings, many new potential users gain exposure to the laboratory. Finally, as 
these testimonials indicate, these information services help laboratories gain and maintain support 
among key audiences. 



Limitations 



The information is often used more to reinforce existing stances than to promote reform. 
Enlightenment about new approaches and awareness of developing issues are potential products of this 
kind of information service, but are also much less likely to be requested by policymakers. It is true 
that this "ammunition'* role, while helpful to state decisionmakers, may represent an inefficient use of 
scarce resources. 



There was not complete consensus about the laboratories' responsiveness to individual state 
concerns. Among the dissenters were two for whom anything but total attention to state needs would 
be insufficient; this is obviously not realistic given the breadth of the laboratory mission. But several 
state officials did mention that they wished the laboratories would provide more information tailored 
to the state's needs (e.g., implementation of performance standards) rather than to the laboratory's 
own agenda (e.g., restructuring). 



There was an interesting disagreement about the usefulness of the current regional 
configuration. For information purposes, many policymakers said that they counted on the 
laboratories in particular when they needed information about what was going on among their closest 
neighbors, but used the Education Commission of the States or the National Governors* Association, 
for example, to learn about national trends. Others said just the opposite: that they especially 
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appreciated the laboratory's access to the latest information about national trends, and found the 
regional boundaries artificial. In one region, an SEA staff person felt that her mostly rural region 
was relatively uninformed since so many of the laboratory's resources went to the largest, most 
populous state. 

The Regional Laboratory as a Unique Information Resource 

Regardless of their experiences with their respective laboratories, we asked all decisionmakers 
we interviewed to describe what was unique about the lab oratory -that is, what information or 
assistance gap it filled. We wanted to know whether the laboratories have in fact carved out an 
important and distinctive niche as a resource for states. Why would a decisionmaker call on the 
laboratory with so many other resources available (e.g., national associations, his or her own research 
department, nearby universities)? 

The answers were thoughtful and wide-ranging; one person described his comments as ideas 
of what laboratories could do other than what they ate doing, while the others all pondered what they 
would miss if the existing organizations were to disappear. Their responses are clearly related to the 
accomplishments noted above, but extend them somewhat. They address three broad features of the 
laboratories* policy work: 

Political neutrality. Many of the policymakers mentioned this characteristic of the 
laboratories. We noted above that this perception adds considerable authority to the research 
laboratories pass along. Several of our decisionmakers noted that the laboratories clearly work hard 
to maintain this image of objectivity among sometimes competing pressures. In a similar vein, we 
heard from several individuals about the sense of stability that laboratories offer states and regions in 
transition. As one SEA staff person said, "When a commissioner leaves and a brand new agenda 
comes sweeping in, the laboratory is the logical place to help you get up to speed." 

Policymakers, for the most part, also noted that the laboratories are unusual in their 
commitment to fairness across their client base. One deputy commissioner from the Midwest 
contended that the large national education associations that are also capable of providing information 
and assistance are much too strongly allied with the h power states'* of California, New York, Florida, 
and Texas; he felt these other agencies had little time for him and his staff. 

This impression of the laboratories as a nonpartisan resource may explain why they are called 
on frequently to act as facilitators (and, as the next chapter will discuss, as conveners). They may be 
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asked to help policymakers not only by providing materials but by creating a sounding board tor ideas 
and their potential consequences. A state board of education director said, "The lab is particularly 
good at helping policymakers think through the relationship between policy and practice-which is not 
a talent the government usually attracts to itself!" 

Breadth of knowledge base. Although there was some difference of opinion here, the 
laboratories were often praised for their grasp of state or national trends -or most often both. 
Policymakers reported that the laboratories are an unusually good source for new contacts on 
particular issues, since they seem to support nearly infinite webs of networks. As noted above, the 
information was typically applauded as being timely, comprehensive and-significantly-based on 
practice rather than "only on research. " This fact was mentioned frequently during the interviews: it 
is important for decisionmakers to have access to the latest news on what has worked and what 
hasn't. 

Like some of the R&D activities discussed . in the previous chapter, some of these information 
exchanges draw on multiple knowledge bases that state officials say they can't find elsewhere. For 
example, one SEA staff member recalled turning to the laboratory when the agency was restructuring 
because the laboratory was the one organization likely to understand all the dimensions of the 
problem. "From [the Council of Chief State School Officers, for example] you can get all you need 
to know about how things are organized structurally, to make things run smoothly. But no other 
agency knows both the administration side and the curriculum side. " 

Finally,, one former commissioner described the laboratory knowledge base as having been 
"always at the forefront of national trends, even going back to the days of the original Effective 
Schools stuff." Interestingly, two state officials contrasted this sharply with the information available 
from universities, which they say is difficult to obtain and often out of date. 

Responsiveness and efficiency. This was a common theme among the satisfied information 
recipients we interviewed. One noted, "Unlike other organizations you might ask for help, with the 
lab you get the feeling that this is their job, and you get their full attention." The information is 
timely and generally written to be accessible to a lay reader This is a service that many state staff 
claim they could never efficiently supply themselves. One respondent from a rural state noted that 
some of the bigger states might well prefer to get the money directly-they have the infrastructure to 
distribute it well-"but we wouldn't have the slightest idea what to do with it. " 

In describing the distinctive gap that the laboratories fill, one SEA staff person wondered who 
else would have the time to put together a document that synthesizes the advice of experts, past 

47 

60 



BEST COPY AVAILABLE 



research, articles, publications she never sees, etc. Or, as summed up by an SEA program director, 
• Who else would come to Carson City? Somebody from Washington D,C? I don't think so." 



Other Information Events and Products 

We also studied a set of activities, focused on research and practice more than policy, that 
shows the most public face of the laboratories: they encompass a range of more structured events 
than either the tailored syntheses or the technical assistance that accompanies R&D. Many of them 
more or less fit the definition of the much-maligned workshop -a reputation which we examine further 
below. 

Here we tried to select one activity from each laboratory that represented a relatively brief 
encounter. Generally speaking, these activities present an audience with specific information about 
promising practices in an attempt to change some aspect of behavior, and they are relatively time- 
limited rather than long-term commitments with follow-up. However, there are also important 
differences among the events we investigated. They were not standard in duration, intensity, or the 
degree of match with the audience (e.g., some audiences were self-selected and some not). Some 
activities were clearly designed to promote awareness about a particular topic rather than to influence 
teacher behavior in the classroom or administrative practice. Finally, some of these activities are the 
result of long-term R&D activities while others stand alone. 

For each activity in this category, we surveyed a random sample of participants from lists 
given to us by either laboratory staff or event organizers. In all, we sent surveys to 448 activity 
participants; after two rounds of follow-up, 314 of these had completed and returned the survey for an 
overall response rate of 70 percent. For further information about methods and a copy of the survey 
(which was slightly tailored to each event or product), refer to the Appendix. We did not conduct a 
survey for NCREL's teleconference series because the laboratory was already developing its own 
survey instrument. We made suggestions for additional questions that could be used in our analysis, 
and some of these questions were included. Unfortunately, at report-writing time, the response rate 
was low, so we could not include survey numbers here. However, the NCREL survey results tend to 
mirror the trends found in our surveys. 
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FWL's Awareness Workshops 



FWL staff members present workshops in school districts and at conferences to teachers, staff 
developers, and principals in rural schools. Topics include thematic instruction, alternative 
assessment, and teaching English learners, and take advantage of interactive formats such as panel 
discussions and symposia. 

Laboratory staff have made presentations at several conferences including the National Indian 
Education meeting, the Rural Education Association meeting, and the Stanford Portfolio Conference. 
By making presentations at conferences, they are able to get feedback on their ideas from teachers and 
administrators* find out about work going on in different school districts, and publicize the laboratory 
as a resource on these topics. 

McREL's Content Standards and Benchmarks Book 

This activity evolved from the laboratory's previous work on the development of authentic 
tasks. During that project, laboratory staff discovered that teachers, schools, and school districts 
needed appropriate information about content standards in order to develop performance tasks-and 
that they were unsure how to synthesize information from the national reports. 

Starting in 1992, laboratory staff reviewed reports issued by national groups, identified the 
implicit and explicit (earning outcomes in the reports, entered them into a database, and developed a 
comprehensive synthesis of national work in this area. 

The resulting 269-page report, The Systematic Identification and Articulation of Cont ent 
Standards and Benchmarks , reviews content standards and benchmarks in science, mathematics, 
history, geography, communication and information processing, thinking and reasoning, working with 
others, self-regulation, and life work. Between January 1994 (when it was published) and April 
1994, the benchmarks book had been ordered by over 280 individuals, schools, and school districts. 
Intended for use by teachers, principals, curriculum developers, local superintendents, other district- 
level administrators, and local school board members, this compilation is most often used to guide 
discussion among those who are involved in developing or refining their own content standards. 



49 

62 



BEST COPY AVAILABLE 



NCREL's Vtdeocon Terence Series 



The teleseries as a whole is an important pan of NCREL's program, since this laboratory 
hopes 10 be a leader in the field of telecommunications in education. The series that was the subject 
of our investigation is the second of NCREL's two teleseries, entitled Schools That Work: The 
Research Advantage . The series was designed and developed by laboratory staff, including experts in 
technology, the content areas, and school change. Staff also worked with three sets of partners: the 
Public Broadcasting System (PBS) as a partner and consultant in production, marketing, and 
dissemination; content partners, mostly from universities; and SEAs, as staff development providers 
and disseminators. 

The teleseries itself was designed as an eight-part series for use in staff development (or by 
individuals), with each part focusing on a particular topic related to the national goals. These topics 
included math, science, reading and assessment, early childhood education, drug education, school to 
work transition, and integrated social services. The series was designed to be integrated into an 
ongoing staff development program: subscribers to the series were also sent specially prepared 
guidebooks. In some cases, live interactive conferences were held at specific down link sites during 
the broadcast with local facilitators. In other cases, teachers observed the videos of the broadcasts in 
groups with facilitators who stopped the tapes to lead discussions. Others just watched the tapes with 
no discussion. According to the NCREL survey data, over three-quarters of the region see the 
information on video tapes at their school. 

NE/Igs Making Change Game 

Now a NE/1S activity, the Making Change Game was originally developed by an affiliated 
organization, the NETWORK, to serve as an enjoyable, three-hour simulation of the change process. 
According to laboratory staff, the need stemmed from an accumulation of comments from 
practitioners asking for help on a variety of topics-all of which had as their basis a need to 
understand the change process. 

The knowledge base underlying the game consists of three strands. The first defines how 
personalities affect the adoption of new ideas-the notion of "adopter types. " The second comes from 
the Concerns Based Adoption Model (CBAM), which builds on four observations about change: that 
it is a process, not an event; that it is made by individuals first and then institutions; that it is a highly 
personal experience; and that it entails developmental growth in knowledge, skills, and feelings. The 
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third strand of research is the basis for the specific "moves" made in the game: the supports that are 
needed Ui facilitate the change process in the schools. 

Although the subject of the simulation is educational equity, the lessons are generic and can 
be applied to other innovations. After playing the game, in the debriefing phase, players generalize 
the lessons of the game to their own situations. The game may be bought or rented from the 
laboratory, where a staff member will either facilitate it or train a local group leader. Although it is 
impossible to tell how many people have played the game, according to the Game TrackingJarm . 
2,000 people have played it with laboratory trainers since February 1992. 

NWREfs C l**cmflm A ssessment Training 

Classroom Assessment Training at the Northwest Laboratory is a major activity, designed to 
provide teachers and others with information about performance assessments and how to align them 
with classroom instruction. The assessment workshops cover a wide variety of content areas but 
focus mainly on writing. The primary goal is to refine the use of assessment in the classroom, so that 
ultimately the teacher can help the students assess themselves and take control of their own learning. 

The two- to three-day workshops are open to anyone who can afford the fee of $300 per 
person. The laboratory tries to encourage groups or teams to attend by offering a discount on the 
rates. The format includes large- and small-group presentations and a number of hands-on activities. 
General topics include assessments in writing, portfolio assessments, and performance assessments, all 
of which are framed by the "Six Trait Model"; the traits consist of ideas, organization, voice, words, 
fluency, and conventions. The training is organized around this model, and feature topics such as 
scoring rubrics, the writing process, strategies for introducing and using the mode,-along with 
specific instructional activities and grading guidelines. The assumption that guides the training is that 
students will become better writers by learning the six analytic traits. 

Workshops vary in size from as few as 20 to as many as 100 participants. According to data 
provided by the laboratory, in 1993 NWREL staff trained more than 8,000 teachers and 
administrators through inservice activities, training workshops r and national training institutes. 
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RBS's Urban Learner Framework Presentations 



The Urban Learner Framework presentations are short "stand-up" sessions hy the director of 
the Urban Education program of the laboratory. Based on an extensive literature review and 
conceptual framework, these presentations to teachers, administrators, and staff developers deal with 
educating at-risk students from urban environments; the focus is on cultural diversity, learning, and 
motivation. 

The presentations focus more on the strengths and resilience of the urban learner than on 
weaknesses or deficits. The trainer is seeking to move from a deficit model to a consimctivist model 
of learning, and has based the presentations on sociocultural and cognitive research. She hopes even 
in these brief sessions to introduce important concepts that will form the basis for more long-term 
working relationships with teachers. 

This framework introduction is often presented as part of a larger gathering or conference, 
and may or may not include showing the project video, in which a fictional urban student named 
"Russell" illustrates the topics covered in the workshops. 



SERVE's Reducing School Violence Workshops 

SERVE's "School Safety and Emergency Preparedness: Reducing School Violence" 
workshops offered teachers and principals awareness-level training in school law, crisis management, 
and instructional techniques related to reducing violence in the schools. The training was developed 
from a popular 1992 SERVE "Hot Topics" document on school violence. 

In November 1993, the Mississippi Department of Education asked SERVE to help it develop 
a workshop on the topic to meet the expressed needs of Mississippi school districts. SERVE worked 
with the state department and consultants to develop the one and a half day workshops* eight of which 
were held regionally in January and February of 1994. Districts sent t^ams of two to the training, 
usually an administrator and a teacher, who then went back to their districts to share the information. 
Participants also left with a list of consultants and organizations-including SERVE-to contact for 
more assistance, which some have done. 
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SEPL's Leadership fo r Change 



The objective of Leadership for Change (LFC) is to document and synthesize the best known 
practices for implementing school reform and to provide those undertaking reform with the necessary 
training and tools. The LFC staff identify the activity's focus on the implementation of change as 
unique. They say that resources designed to help people plan for reform are widely available, but 
that very little of the available information or training actually focuses on implementing change. The 
two and a half day workshops are targeted to professional development specialists, who will then train 
anyone who is. or plans to be, involved with the change process. 

The training component of LFC consists entirely of the workshops. These typically include 
20-30 participants and three to four trainers. The trainers walk the participants through selected 
portions of a large box of materials assembled to equip the participants with everything they need to 
conduct their own workshops. Over 300 individuals have gone through the LFC training in the last 
two years. 

SEPL's Country Stars 

Country Stars is a series of forums that feature presentations by rural educators who have 
been identified by the laboratory has having developed promising practices. With only one exception 
that we are aware of, all of the forums take place as parts of larger conferences and meetings 
sponsored or cosponsored by one of state associations that serve as partners. With these 
presentations, the laboratory staff hope to accomplish several objectives: to facilitate networking 
among small rural schools; to disseminate information about rural school improvement strategies; and 
to provide an opportunity for state education leaders to respond to the new ideas offered. 

Laboratory staff and participating "stars 1 ' provide literature reviews and profiles of the 
promising programs. The stars themselves usually present their own material, although occasionally 
outside speakers will offer sessions on particularly popular topics, such as technology and distance 
learning. 

Results 

Three-quarters of our survey respondents remembered attending the laboratory workshop for 
which we surveyed them or receiving the materials sent to them. Of the 24 percent who said either 
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that they did not attend a workshop (even though their name was on the attendance list) or that they 
didn't remember attending, nearly half were answering with regard to an activity that was part of a 
much larger conference. Ninety-five percent of respondents who had received a product remembered 
that they had. We base the discussion below only on information from respondents who remembered 
participating in the specific activity or receiving the product. 

Participants and recipients praised the quality of the presentations and materials. Of those 
who remembered attending a laboratory activity or receiving a product, over half gave a "very high" 
rating to the presentations (59 percent) and the materials (62 percent), and 36 percent rated the 
presentations and materials as "somewhat high." Fewer than five percent of participants were not 
impressed by the presentation they attended. In fact, 94 percent of those who attended one activity 
gave the presentation a "very high" rating. In addition, 43 percent felt that the workshop presentation 
and materials was unique in some way. These high ratings may be explained, in part, by the 
participants* perceptions of the information's accuracy and usefulness. Nearly all (97 percent) of 
participants and product recipients said they felt the information that was presented was accurate and 
complete (65 percent said very complete, 32 percent said somewhat complete), and 90 percent felt the 
information was useful. 

Because most of our data were collected through surveys, little anecdotal evidence is 
available. However, we had the opportunity to speak with some activity participants about their 
involvement, and their responses were similar to those found in the surveys. One participant said of 
the presentation she attended, "The outcome was very favorable... it [the activity] is very good, it's 
not boring. It is well thought out and low key. The materials and video are helpful." Another 
added, "This is some of the best work relative to how we can better prepare today's child." A 
participant in another activity said, "1 feel that the quality of the sessions was excellent. The lab did a 
great job assembling the material-there were a lot of good examples ... a good balance between 
theory and practical application." 

Participants indicated a fairly high level of use of the lessons learned in the workshops and 
the materials. Nine-tenths of the participants indicated that the information they received in the 
laboratory activities was useful to them. In fact, two-thirds of workshop participants said the 
presentations and materials were more useful than activities they had attended offered by other staff 
development or assistance providers. The information was useful, primarily, for keeping the 
participants and their institutions aware of developments and activities in education and giving the 
participants resources for helping others. Over half of participants indicated the information made 
them aware of developments, and provided the information to others to a "great extent" and over one- 
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third said to "some extent." A significant proportion of participants (39 percent) said the information 
encouraged Uiem to try something new on the job. 

For the most part, these workshop participants are passing on the information to their 
colleagues although the workshop presenters are not pushing them to do so. A full three-quarters of 
workshop participants said they had passed the workshop information and materials on to others 
inside their school or agency, and half (54 percent) said the receivers were using the information for 
their own purposes. Significantly fewer shared the information outside their own workplace: only 26 
percent said they had passed the information on to others outside their school or agency. 

The information contributes to the conversation among educators. One laboratory staff 
member reported that "Now we are getting very specific requests for help in implementation as 
opposed to requests for information about what it [the topic] is." A participant at NWREL's 
Classroom Assessment Training reported on the survey, "We are now using the six traits in most 
classrooms in the district as both a teaching and assessment model. This has given us a common 
language in writing instruction. * A staff developer who received the Content Standards and 
Benchmarks book described its utility in this way: 

We've undertaken a major transformation without providing enough resources to teachers. 
We need to give them examples of standards; show them how to create benchmarks from 
those standards; and show than how to write a performance assessment from a standard. I've 
used the report as a reference in developing benchmarks. 

Workshop participants generally agree that the workshops were a good use of their time. 
Nearly all (94 percent) of workshop participants felt the presentations they attended were worth their 
time. This was particularly true for workshops that lasted more than one day. 3 Nearly 60 percent of 
participants who attended multi-day workshops said they received the right amount of information in 
the right amount of time, compared with 42 percent of those attended shorter activities. 

The cost of the product or service does not affect the likelihood of behavioral change. 
Activities we examined ranged from no cost at all to $400 per participant. There were no significant 
differences across the group in subsequent reports of use, change in practice, or sharing of materials 
with others. Not surprisingly, attendees at the expensive workshops were more prone to remember 
attending; they also praised the quality of materials and the presentations more highly, and were more 
likely to see the activity as unique. 



2 Workshops considered for this survey lasted between 2 hours and 3 days. 
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The region becomes acquainted with the laboratory as o professional resource. Laboratory 
events and products can introduce educators in the region to the laboratories* resources. Over half 
(59 percent) of workshop attendees and product recipients indicated that they had not been familiar 
with the laboratory in their area. Three-quarters of this population indicated that attending the 
workshop or receiving the product encourages them to contact the laboratory for additional materials 
or assistance. One workshop presenter said, * workshops... increase the visibility of the lab. 
Presentations are an essential part of what 1 do. M The high marks on the presentations and materials 
may enhance the laboratories' reputations as staff developers, service providers, and disseminators. 

Workshops also allow the laboratory to build relationships with other service providers such 
as conference organizers, technology developers, local workshop presenters, and other regional 
laboratories. One laboratory cited as an accomplishment that the workshop has become a regular 
agenda item for several state conferences. A staff member gives the following evidence of an 
improved relationship with one association that organizes an annua! conference, starting with the first 
year: 

We talked conference organizers into letting us have the only open block in the schedule of 
the annual summer meeting. It tuned out that we competed with the annual fishing 
tournament and had a hard time' getting anyone's attention. We're doing better now and 
getting more established and recognized as a resource, but it's still a struggle. We don't 
compete with the fishing tournament any more. 

Laboratory stqff have the opportunity to interact with the field. Workshop presentations can 
provide a useful forum for sharing information as they typically generate discussion abound the topic 
presented, in most cases, attendees have the opportunity to ask questions about the topic presented, 
and to share their efforts and frustrations. These discussions allow laboratory staff to learn more 
about the region's capacities and needs, which can be used to focus their current presentations or 
develop additional resources to address further issues. Staff of one activity said, "Presentations at 
conferences allow us to find out about work that's going on in different school districts, bounce ideas 
off teachers...." Workshop attendees also benefit from the discussion and comments-they hear their 
peers' triumphs and dilemmas and can use the information in their own situations. 

Short-term products and services are connected to the more in-depth work of laboratories. 
Even the shortest events considered here are offshoots of larger laboratory ventures, with the result 
that they can incorporate a broad and deep knowledge base. Participants do not necessarily know 
this, but we believe some of their positive comments reflect a recognition that there is substantive 
depth behind the presentations. The Urban Learner Framework, for example, represents several 
months of conceptual work on the part of program staff who sifted through research to build a 
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framework that would underlie their entire program of assistance. An organizer of one of the 
presentations of this framework commented: 

The lab provides the cutting edge, leading technical assistance. They have pretty much 
compiled and synthesized the latest information, and so they were very excellent facilitators in 
trying to connect pieces of [my] organization. They get you to think on a broader level. 

Similarly, the assessment training offered by NWREL received very high ratings from survey 
respondents and is based on several years of field research. The Making Change game is another 
example where a limited time commitment and a light touch are merely the vehicles for presenting 
ideas that are grounded in years of serious research. 

The connection between single events or products and ongoing R&D can work in the other 
direction as well. Several laboratory staff members told us they continually weigh the value of 
making presentations but that what they learn about the current work and concerns of teachers does 
help inform their R&D activities. 

Sometimes a planned connection between research and an information event does not 
materialize. Leadership for Change was originally intended to build on a qualitative field study in 
five sites wine leaders were initially entering the school improvement process or were already under 
way in their efforts. The idea was to document factors supporting or impeding change in schools and 
to build this knowledge into the training. However, conclusions have been slower than expected in 
emerging from this study, and instead the LFC training builds on the large body of existing research 
on school change-some of which was conducted by laboratory staff. 

Limitations 

Although the workshops and products are popular, there are limits to their ability to inspire 
broad-based reform efforts. 

Workshops and products may provoke individual change, but this rarely filters through to 
organizations. Workshop participants generally attend the activities on their own or with a few 
colleagues. Rarely is a majority of the school present. This phenomenon becomes clear in our 
survey results-individual use of the information is much higher than institutional use. Although the 
NCREL evaluation data on the teleconference suffer from a Limited initial response rate, the data 
show that teachers were twice as likely to say the video's ideas and skills fit into their personal 
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professional priorities than into the school's priorities* Similarly, our survey data showed that the 
information derived from other workshops or products was rarely used at the institutional level. At 
the institutional level only 19 percent of respondents use the information to initiate a new program or 
policy, and even fewer (6 percent) use the information to determine whether to terminate an existing 
program. Approximately one-third of the respondents said their institutions use the information to 
support current policies or to generate awareness of the topic. These data indicate that workshops and 
products are unlikely to lead to lasting behavioral changes in schools. The odds that a workshop will 
catalyze change at either the personal and institutional level increase slightly under the following 
conditions: 

• The workshop lasts for more than one day 

• The information is targeted for the attending audience 

• The workshop is geared toward further client inquiry instead of focusing on 
information transmission. 

We elaborate here on each of these conditions. 

Although workshop attendees and product recipients generally think the services are a good 
use of their time, nearly half say the information presented "only skims the surface. " 
Approximately 45 percent of respondents indicated that they needed more information, interaction, or 
follow-up to help themselves or their organization. One participant in the Making Change Game said, 
"While fun, useful, and interesting, it is still just a game. I suspect, therefore, that it was basically a 
superficial experience/ 

In our sample of activities, the lack of workshop time (e.g., two- or three-hour slots) tended 
to be the primary factor in determining whether participants felt they had received sufficient 
information on the specific workshop topic of whether they felt they needed further follow-up 
information. Among participants who attended multi-day workshops, 41 percent felt they needed 
additional information compared with 58 pe> lent of those who attended shorter sessions. One 
workshop organizer discussed the trade-off between depth of the activity and the available time, 
saying, "It was basically an introduction of concepts. I wish it could have been more in depth, but 
there was a time problem in fitting in workshops." A participant in a different activity said, "My 
assessment of the activity? What can you accomplish in a half-hour?" 

In addition to the workshop's duration, another issue-whether the workshop provided in- 
depth information-figured into the attendees' perception of the workshops. Among those who 
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attended multi-day workshops. 41 pe L c«nt were attending sessions that offered in-depth information on 
the topic/ Nearly three-quarters of ttk -e attendees said that their workshop had "provided the right 
amount of information in the right amount of time" and that they did not need additional follow-up. 
In contrast, 49 percent of attendees of sessions that provided a subject overview said the same. All 
workshops in our sample that provide in-depth information last for over one day. Over half (58 
percent) of attendees in shorter workshop sessions indicated that their workshop just "scratched the 
surface" and they needed further assistance. 

Not targeting the information to the specific audience can reduce the client impact In our 
sample, a potential mismatch of information and audience occurred in three ways: (t) workshops were 
conducted by laboratory staff for a specific audience (e.g., teachers in a particular district) but the 
information did not address the audience's needs; (2) workshops were held as part of a larger 
conference and participants representing various backgrounds and needs selected the session (e.g., 
Country Stars); or (3) workshops were pilot tested on an audience that differed from future 
participants in important ways. Leadership for Change is an example of tins last circumstance: the 
field-test participants were superintendents and principals, who responded very positively to the 
training and materials, but whose needs and agendas probably differ from those of the staff developers 
who are the usual participants in the training now offered. We also found that some product or 
workshop developers launched their offering with only a hazy idea of who would use the information 
or what their needs, capacities, or preferences would be. One developer expressed enthusiasm about 
our plan to survey educators, saying that he occasionally wonders how anyone would use what the 
laboratory is offering. 

In those cases where the information was targeted to its audience (e.g., the Mississippi Anti- 
Violence Workshops), participants were much more positive about the presentations and materials. 
Nearly three-quarters (71 percent) gave the materials a "very high" rating (compared with 54 percent 
of those where the information may not have matched their needs), and 66 percent did the same for 
the presentation (compared with 51 percent). 

When the information is tailored for its audience, the audience is more likely to use the 
information to take action. Nearly half (46 percent) of these attendees said that, to a great extent, the 
information encouraged them to try something new on the job (compared wirii 34 percent) and half 
said that, either to a great extent or some extent, the information had convinced them to stop what 



3 Activities were designated as in-depth or subject overview based on data collected during our site 
visits to the laboratories. In-depth activities include: ContenlSiandards and Benchmarks . Classroom 
Assessment Training, and Leadership for Change. All other activities were considered overviews. 
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they were doing and try something new (compared with 3 1 percent). And finally, attendees of 
sessions where the information was more closely targeted were more likely to pass on the information 
to others in their agency, and to feel the activity was more useful than what they have received from 
other professional developers or technical assistance providers. 

Conclusion 

The popular perception of regional laboratories typically features workshops and document 
production as central to the laboratory mission. Although educators interested in reform-especially 
laboratory staff—are coming to understand the importance of longer and deeper involvement by 
education professionals in their own learning, it is clear that these short-term events and products still 
have a place in the laboratories' portfolios. 

Similarly, the individualized responses to policymakers' inquiries-wfaether through research 
syntheses or task force development— fill an important role in the laboratories' repertoire. 
Decisionmakers in the region trust the quality and neutrality of the information they receive, and 
appreciate the prompt service. The laboratory, in turn, is able to enrich its ongoing dialogue with the 
field, learning about current constituent issues and needs as it maintains a supportive role in the 
region. 

Despite the prevailing wisdom-"we know that workshops don't work"- "one-shot" 
workshops can be effective for a number of purposes. Our survey data and interviews with 
laboratory staff indicate that workshops are popular among participants. By their own reports, 
participants are using the workshop information and passing it along to colleagues both inside and 
outside their workplace. However, according to these data, one-shot workshops are not an effective 
tool for initiating and sustaining school reform efforts. 

While it is true that participant satisfaction is in itself no indication of potential change, 
keeping the customers happy by providing new approaches to difficult issues is not a trivial 
accomplishment. As we have seen, it acquaints them with laboratory services, provokes them to 
share what they have learned, and strikes them as a worthwhile use of time in their busy days. These 
brief events are not intended to revolutionize schools, but rather to raise awareness and to furnish 
tools for problem solving. Expecting more from such short encounters would be unrealistic. 

Laboratories are constantly faced with decisions about priorities, and laboratory staff say they 
seek a balance between short-term events and more sustained improvement efforts. A particularly 
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candid laboratory manager told us Chat there is always a temptation to do too many workshops. When 
you give people information they value in response to a request, they lavish you with praise, and, he 
notes, "It's immediately more rewarding than trying to work through a more systemic approach. " 
This is particularly true when the request comes from someone you need to care about: a 
superintendent, a legislator, a state board member. He continued, "It's a tough balancing act. It's 
hard to tell the state board or legislature that they [we] don't have time to help. [If we say that,] the 
district or the state superintendent both ask, "Well, what good are you, then?" 

Whether or not these activities represent a good investment on the part of the laboratories 
depends on desired outcomes. If research and practice knowledge is to affect behavior in the 
classroom, it can only be through a prolonged relationship that is a bona fide two-way exchange. The 
H one-shot workshop," as we know, is largely a monologue that denies teachers the opportunity to 
engage with the material and reconstruct it for themselves. In spite of this, exposure to new ideas 
contributes to reform even when its effects are subtle: to enlighten and to change the conversation 
about problems and solutions. 
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IV. TECHNICAL ASSISTANCE TO BUILD CAPACITY 



Technical assistance that supports product development and the creation of model instructional 
programs (NCREL's Strategic Reading Program) or change processes (NWREL's Creating the 
Future) is shaped by the requirements of the development process. In these activities, assistance is 
designed to facilitate testing, implementation, or use of a laboratory product or program. In this 
section of the evaluation, we examine technical assistance that supports the more general goal of 
capacity building in institutions that, in turn, focus on improving schools, school districts, or other 
organizations. Compared with assistance in support of development, this assistance is more explicitly 
focused on client needs and local circumstances. Consequently, it requires more ad hoc choices about 
specific assistance tasks and allocations of laboratory resources. Not surprisingly, the assistance 
results in less clearly identifiable outcomes. 

Activities Studied 

This evaluation looked at five examples of technical assistance to support capacity building. 

McREL's Missouri School Improvement Program 

The goal of the Missouri School Improvement Program i as to redesign the state accreditation 
system so that it generates better assessments of school quality and contributes to statewide school 
improvement efforts. The project evolved through several years of conversations among a senior 
member of the laboratory staff, an official in the Missouri state department of education, and a 
consultant from the University of Missouri. The challenge was to create a new system for accrediting 
school districts. The new system would replace a traditional input model, which had been in place 
since 1930, with a system that would use more appropriate indicators of school quality and that would 
rely on a review process more likely to support local improvement initiatives. In addition, state 
officials and local superintendents wanted a system that would not reward larger, more affluent 
districts at the expense of smaller, often poorer districts. The challenge of creating the new 
accreditation system was made even more difficult by the existence of a well-entrenched network of 
state supervisors who wanted to preserve the old system. 
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The laboratory's assistance concentrated on addressing the technical issues of the accreditation 
system, It also included considerable attention to strategic planning for statewide pilot testing of the 
system and training of state department officials and others who were to be involved in the new 
process. Implementation of the new accreditation system began during the 1991-92 school year, with 
all Missouri school districts slated to have been classified under the new system by the end of the 
1995-96 school year. 

SEDL's Stren gthening Science Partnerships 

The laboratory's long-range goal for this project is to improve science education for at-risk 
students by helping to forge partnerships between science-rich resources (e.g., museums and science 
centers) and schools and school districts. In practice, laboratory assistance consists of working with 
individual museum and science center staff on projects ranging from the evaluation of new materials 
to professional development programs for elementary and secondary science teachers. Typically, 
laboratory staff have used the initial assistance activities to market other laboratory services and the 
prospects of long-term relationships through which the laboratory could contribute to the training, 
outreach, and dissemination capacities of the various science-rich resources. 

RBS's Support for Philadelphia SrJioolwide Protects 

As an extension of its work with the Pennsylvania state department of education's Chapter 1 
office, the RBS State Assistance Program works with the Philadelphia Public Schools' Schoolwide 
Projects Office and die Office of Accountability and Assessment. Since 1990, laboratory staff have 
completed a number of case studies of various facets of individual Chapter 1 programs in the 
Philadelphia public schools. The results of these studies have been made available to the district for 
dissemination throughout the system. Workshops and consulting services have also been available to 
help district staff on specific tasks related to program administration and student assessment. We 
focus here on those parts of the laboratory's work whose goal is to enhance the Schoolwide Projects 
Office's capacities to provide technical assistance to schools and to disseminate research-based 
information throughout the system. 



63 

76 



BEST COPY AVAILABLE 



PREL's R&D Cadre 



The R&D Cadre brings together 14 representatives from the far-flung entities served by the 
laboratory, regional institutions of higher education, and private schools to conduct four applied 
research studies on topics that are critical to the region. The goal of this activity is to generate 
information and prepare reports to guide legislators and administrators in decisions about educational 
reform. This activity also serves as a primary source of information about regional needs for the 
laboratory. Two of the four research projects examine policy issues, and two are intended to address 
curriculum issues. The topics include: 

• School finance and facilities 

• Equitable access to learning opportunities 

• At-risk factors among Pacific children 

• Home/school learning styles 

Cadre members are nominated by the chief education officers of the entities. Members 
include mid-level central office staff or building principals. Each person is expected to devote 10 
percent of his or her time to the activity. In addition* each entity is expected to establish a support 
group of teachers who are responsible for assisting in field-based data collection. Laboratory 
assistance, which began in 1990, includes financial support for cadre members' travel and other 
expenses, training in research methodology and in training local support groups, help in data 
collection and analysis, and ongoing review of findings and conclusions. Assistance is provided at 
biannual meetings and through frequent electronic communications. 

Tire cadre has completed one publication, The Pacific Region School Finance and Facilities 
Study t whxh has been widely disseminated and is considered by some to be the best source of data on 
the region. 

FWL's Marin CHv Families First 

In Families First, laboratory staff bring two ambitious goals to their work with a small, 
nonprofit social service agency that serves a poverty-ridden housing project in Marin City, California. 
The first goal is to continue work on the development of a complex, early intervention model for 
family and community services. The second goal is to work through the social service agency to 
bring together other social service agencies serving the same community. It is the second goal that 
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leads us to include this activity in this chapter of the report. We did not include it as a development 
activity because the model building process is less formal than in the activities we discussed in the 
earlier section of our report. 

Laboratory staff assist agency staff in several ways. Regular meetings with two case workers 
and staff from other agencies include review of decisions about services to individual families. 
Laboratory staff augment these sessions by providing information and materials on infant health and 
nutrition, parent-child interactions, child development, substance abuse counseling, child care, 
employment training for adults, and the home visiting process. There is also training in the case 
management approach to working with families and how to help families gain access to the resources 
they need. In monthly meetings with the agency director, laboratory staff" oiler advice and guidance 
on a wide range of operational concerns. One of the more pressing operational concerns is funding. 
Foundation support for the agency ceased shortly before our visit, and most of the agency's attention 
as well as the laboratory's assistance focused on garnering funds for the agency. 

Assistance in Support of Capacity Building as a Category of Laboratory 
Activities 

The activities that we have included in this category share five characteristics. 

Attention to Development Tasks and Changing Perspectives 

These activities blend help on specific tasks with efforts to develop or change perspectives on 
the nature of the work to be done in the organizations. Laboratory staff who work with the R&D 
Cadre train cadre members to conduct research and help in all facets of the research projects that span 
the Pacific region. At the same time, laboratory staff spend considerable time helping cadre members 
develop an understanding of the value of research in guiding improvement efforts and in setting 
policy. Laboratory staff who assisted in the Missouri School Improvement Program spent almost as 
much time helping their recipients of the assistance think about and negotiate their way through the 
political mine field that surrounded the initiative as they did in helping develop a new set of indicators 
and a site-based peer review process. In helping museum and science center staff conduct individual 
training and professional development activities in Strengthening Science Partnerships, laboratory staff 
have tried to encourage development of a broader understanding of what good training and 
professional development should look like and how they can become integral components of museum 
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and center outreach activities. Laboratory work in Marin City Families First grounds assistance in 
developing specific case management skills in the larger task of helping service providers understand 
the need to collaborate with other agencies and to become advocates for individual families. 

Task Specific Versus Research-Based Assistance 

AU of these activities, like the other laboratory activities we studied, draw upon research. 
However, unlike the development activities we discussed above, assistance in the activities discussed 
here is guided as much— if not more— by laboratory staffs understanding of what needs to be done 
than by research findings. To be sure, laboratory staff often assert the research base for what they do 
and for what they advise others to do. As one laboratory staff member said in discussing efforts to 
overcome the potential recipients' reluctance to laboratory assistance: "We must be careful not to 
come out with a position unless it is research-based." Nevertheless, because much of the assistance 
provided in these activities is ad hoc and task specific, there is no easily discernible core of 
knowledge that consistently guides the laboratories' assistance. Instead, recipients perceive that 
laboratory staffs understanding of research adds credibility to their advice. Our overall observation 
is that much of the assistance, including assistance that seems particularly effective, is driven by 
thoughtful pragmatism. A description of the Missouri School Improvement Program contained in a 
1987 external evaluation report applies, in varying degrees, to all five of these activities. 

Its important processes and segments were seldom expressed in advance in the form of linear 
plans replete with events, timelines, and milestones. Rather, it evolved in a disjointed and 
certainly incremental fashion with 'milestones' linked more by hopes and hunches than by 
data-based projections (Walker, 1987, p. 1-2). 

Flexible Allocation of Resources 

The third characteristic of the assistance provided in these projects has been the availability of 
laboratory resources to support and augment the assistance. Laboratory staff have been able to 
marshal a variety of resources when they are needed. Often what is most important is the timeliness 
of the resources, not the actual amount. Indeed, the amounts of resources are relatively small, but 
they count because they are available when needed. In the words of a staff member in a science-rich 
but resource-poor museum: "The laboratory was like deep pockets to us." In addition to funds and 
other tangible resources, the less tangible resources of laboratory staff time and energy were almost 
always available when they were needed. 

66 

70 



BEST COPY AVAILABLE 



Assistance to Small Numbers of People 



The third feature common to these projects is that they provide direct assistance to small 
numbers of people. Marin City Families First staff worked primarily with a local program director 
and two family advocates. Assistance to the Philadelphia Schoolwide Projects Office and the Office 
of Accountability and Assistance mainly involves three people. The R&D Cadre includes 14 
participants. The assistance in the Missouri School Improvement Program concentrated on two 
people, although some laboratory-sponsored meetings and training sessions included many more 
individuals from around the state. 

i 

Intense Assistance Over Long Periods of Time 

The fifth feature that is common in the assistance in most of the projects included in this 
category is its relatively long duration and relatively high intensity. The assistance in creating 
Missouri's school accreditation program began in the early 1980s and continued for about ten years. 
Along the way, there were periods in which laboratory staff met with state department staff and 
consultants every month for two or three days. The R&D Cadre has been in existence since the later 
1980s, although the individual participants who represent the various entities in the region have 
changed. Laboratory staff meet cadre members for very intensive sessions twice a year and remain in 
regular contact about the various research projects in the interim. In Families First, laboratory staff 
meet with social service agency staff for several hours at least once a month. 



Impact and Effectiveness 



The primary impact of four of these projects was on the individuals who were the direct 
recipients of the technical assistance. Only the Missouri School Improvement Program appears to 
have resulted in significant organizational changes as the Missouri state department of education 
implemented the new school accreditation system during the 1991-92 school year. None of the four 
projects that focused on education issues, including the Missouri School Improvement Program, has, 
as yet, had much of an impact on schools, according to the data we were able to collect; such impact 
may exist, but it seems unlikely to be large. 
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Participants' Perceptions That the Quality Qf Technical Asslstam* Is High 



As in many of the other laboratory activities included in the evaluation, all of the people with 
whom we spoke had high praise for the services and for the people who provide them. A museum 
staff member commented that laboratory staff "were more thorough and more professional than the 
individual evaluators" she has worked with in the past. She also felt that "SEDL's connections and 
perspective across the five-state region are unique." In Missouri, one of the two primary recipients of 
laboratory assistance concluded that "we would not have gotten anywhere without [the laboratory staff 
member]. The other commented: "[His] track record in rural education was important. This would 
not have happened without him." District-level staff in Philadelphia say that: 

RBS devised a reflective piece for schools to use to look at past practices... It helped them to 
see how to broaden their leadership to be more inclusive of the whole staff. The other piece 1 
was excited about was the school support teams... It gave us information about our program 
and how it was operating... They're helpful in terms of having people rethink what their 
strategies are.... [The laboratory staff member] has supported our school improvement 
program office and helped it to grow....RBS understood working with us to help schools 
grow. I'm so appreciative of an RBS who works with us. We grow, too. 

Later, in comparing assistance from RBS with assistance from another organization* she 
echoed another theme we heard in comments about assistance from other laboratories: "We need to 
help and support [schools], but we need people who do not try to dictate to schools what to do. 
That's the success of RBS." 

A number of the recipients also appreciated the technical quality of the assistance they 
received. Museum and science center staff commented on the quality of the science content of Minds 
on the Universe, a laboratory-developed instructional kit. Hie kit includes a "lively set of activities, 
presented in an inviting, adaptable format." Staff also commented on laboratory staffs knowledge of 
teacher training and staff development. In Missouri, state department staff concluded that the 
laboratory staffs knowledge about rural education, educational change, and evaluation were critical to 
the development of the accreditation system. 

Impact on Individual Professional Perspectives 

With some exceptions, the primary impact of the technical assistance provided by these 
projects was to begin to change individual professional perspectives. People may come to appreciate 
the importance of a particular activity or they may come to appreciate the possibilities of doing what 
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they do differently. A comment by a member of the staff in the Philadelphia School wide Projects 
Office was typical of many that we heard: 

They help us to stand outside of our process and examine it. They help us to regain that 
perspective of taking another view of what we are doing. They kind of hold our feet to the 
fire and that's good for us.... They keep us advised about important issues and staff 
development that we wouldn't be aware of, and provide a forum for us to interact with other 
people. 

In addition to appreciating laboratory assistance on specific activities for which they were 
responsible, museum staff say that the assistance helped them to think differently about staff 
development and training. Participants in the R&D Cadre believe that the cadre activities have helped 
them come to see the vahie of research to inform policy and program options. Said one: "The R&D 
Cadre has been a real eye opener for educators on our island, helping them see the importance of 
research studies in helping [us] solve [our] own problems." Of course, seeing the possibility of doing 
something new or doing something in a different way are different from actually carrying out a new 
task or doing an old one in a new way. 

Changes In Individual Professional Behavior and Organizations 

Participants in Families First, the R&D Cadre, and the Missouri School Improvement 
Program report specific changes in how they do their work and some impact on their organizations. 
For example, a family advocate in the Families First social service agency said: 

One family is doing really well. When I started working with them, the parents were still 
separated and the mother gave birth to a child [who tested] cocaine-positive. Now the parents 
live together. The children are in child care and the mother is trying to go back to 
school... .We are getting the kind of respect that hasn't been there before. Other agencies see 
us as the advocates for these families. * 

At the time of our visit, another member of the staff reported that: 

Things have changed in that [the agency] is beginning to have some authentic dialogue with 
social service agencies in Marin County. Recently, several county agencies put together a 
proposal on child abuse services. [The agency] was called in after the proposal had been 
developed. The [agency] staff told the other agencies that it wouldn't work. 1 would say 
that, as a result of [the agency's] persistence, we are seeing some changes. 
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Participants in Che R&D Cadre also saw changes. "This information really helps us convince 
the school board of our needs because we are so far behind others in resources. The facts gel a 
chance to speak for themselves." A participant from another eniiiy observed that: 

The use of the documents (produced by the cadre] is left up to local discretion, so we're 
learning the best way to use the information to influence people. It's the same way with the 
data collection instruments that we have developed as a group-we each have to decide how to 
best work with the support team to collect the data in a way thai makes sense for our own 
entity. 

The most striking example of impact of technical assistance was in the Missouri School 
Improvement Program. As we noted above, this assistance led to the design and implementation of a 
new state accreditation system. Along the way, the process entailed working through a variety of 
technical issues, including setting new standards for quality and determining what kinds of 
information should be gathered to assess the extent to which districts are meeting the standards. The 
process also entailed help in guiding the new program through strong opposition from deeply 
entrenched interests in the state department of education. Finally, the assistance included helping 
prepare several hundred people for their roles in the new process. Individuals with whom we spoke 
offered the following examples of the impact of the new program: 

• The new* accreditation system is viewed by many as the foundation of new state school 
reform legislation, which went into effect in 1993. Under the new legislation, each 
school district must have an improvement plan, with severe sanctions against those 
districts that are not accredited. 

• The role of state supervisors of instruction is shifting from one of compliance 
monitoring to one of assisting district in preparing for the review process and in 
developing the improvement plans. Several state department staff confirmed that this 
new role represents a more general shift in perspective in the agency. 

• There is considerable attention to examining student performance 

• Participation on a review team is an important professional development opportunity 
for teachers and administrators. They learn how to help their districts prepare for the 
review, and there is an opportunity to learn from other practitioners and local 
administrators. 
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L imited Transfer of New Knowledge and Skills to New Issues in New Setting s 



It the long-term goal of these activities is capacity building, then one measure of success is 
not the completion of specific tasks or even the mastery of the skills to complete them; instead, this 
measure of success is whether the recipients and their organizations are able to use their new 
knowledge and skills to solve new problems in new settings. 

When assistance involves people engaging in new activities, one result can be that they 
develop confidence to try more things on their own. Our interviews with participants in the assistance 
activities suggest that some of these activities afforded opportunities for active involvement and that 
there is some possibility for transferring the lessons from these experiences to other settings. 
Participation in the R&D Cadre is an example. As a member of the cadre told us: "Our experience 
with [the cadre's first research project] has given us the confidence to do our own at-risk study at the 
middle school level. " This comment and others that we heard in interviews with the cadre members 
suggest that their active participation in initial research activity, combined with the fact that the 
activity had a positive outcome that was valued in the region, may lead them to try to apply these 
skills to tasks in other settings. A museum staff member who has worked with the laboratory on two 
consecutive annual workshops for science teachers described the transition from learning to doing as 
follows: "They [the laboratory staff] worked very hard in the beginning and then seemed to know 
when we were ready to take over. I'm not sure we thought we could (organize and conduct the 
training], but we Hied and things went very well/ 

These are promising but isolated examples of participants developing skills and understanding 
that they transfer to other tasks and other situations. More often, the technical assistance achieved the 
proximate goal of completing a task (e.g., conduct training, evaluate a program, disseminate 
information). 

Limited Impact on Schools 

Our data suggest that none of the four education -related activities in this category has had 
much impact on schools. Indeed, only one of the projects, Strengthening Science Partnerships, set 
out to have a direct impact on schools. In this case, the laboratory attempted to field test a science kit 
with a group of teachers from isolated rural schools. According to a member of a museum staff, as 
soon as the laboratory began working with the teacher-developers, it became clear that the idea would 
run into difficulty. The teachers simply did not understand their task. They had no prior experience 
with active learning in science; the idea of diverging from the text to teach with open-ended materials 
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was anathema to them. Teacher participation dwindled, and after many visits to try to engage the 
teachers, the laboratory abandoned the effort. The museum staff memher reported that, despite the 
failure of the experiment, the experience opened her eyes to the "challenges of reaching rural 
populations in her own state/ 

Laboratory assistance to the Philadelphia Schoolwide Projects Office involved conducting case 
studies of several Chapter 1 schools that operated schoolwide projects. As a member of the 
laboratory staff noted: "We are in the schools not to make a change, but get information from them. " 
Staff > ' e of the schools felt that they didn't get anything out of the study: 

The report wasn't helpful, but it was accurate. There weren't any suggestions. There wasn't 
any follow-up. It wasn't a tool we could use. There was no big impact, because it didn't 
really make an impression, [A laboratory staff member's] questions didn't make us think or 
help us along the way. 

Staff in a second school felt that the laboratory's report on their school led them to think 
about things that they might not otherwise have thought about and gave them some direction for the 
future. According to the former principal, documenting the development of alternative indicators was 
useful because "it's hard for schools to document things. We don't have time. They helped us see 
the overall picture/ Nevertheless, staff in this school do not attribute any changes to the laboratory's 
work. 

The Missouri School Improvement Program and the R&D Cadre have school improvement as 
long-term goals, but it will be several years before any changes should be expected. In both cases, 
changes that may occur will be only indirectly attributable to laboratory assistance. For example, in 
Missouri there appear to be two possibilities for changes resulting from the new accreditation system. 
The first is that the accreditation process, particularly the requirements for a school improvement plan 
and preparations for the visit by the state review team, will lead to changes in local policies and 
practices. Second, participation on a review team could provide ideas to teachers and principals that 
lead them to make changes. Similarly, gathering information about education in the Pacific entities 
served by the R&D Cadre could be the foundation for changes in schools in that region. The logical 
connections suggested here are not difficult to imagine. At the same time, it requires more optimism 
than logic to assume that changes will occur and that they will be attributable to the technical 
assistance provided in these laboratory activities. From a school-level perspective, the laboratories 
could appear to be engaging in trickle down technical assistance* 
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The Challenges of Collaboration as Process and Content in Technical Assistance 



Much is written today ahout the importance of coordination and collaboration among public 
agencies. These partnerships, so the commentaries and exhortations suggest, have the potential to 
reduce overlap and inefficiency, to increase the quality of services and the number of individuals and 
families who receive services, and to ensure that individuals and families who need services do not 
fall in the cracks between agencies. Experience in these ventures also suggests that they are 
extremely difficult. They threaten longstanding organizational and professional domains, and they 
require substantial amounts of time, energy, and resources to create and implement. 

The difficulties of addressing the problems of collaboration and coordination are confirmed by 
the experiences of these technical assistance activities. The initial vision of Strengthening Science 
Partnerships was that the laboratory would work with science-rich resources and schools and school 
districts to help them form partnerships that would facilitate student and teacher access to and use of 
high-quality science content. Once these partnerships were created, the laboratory would fade from 
the scene. This apparently uncomplicated idea did not work for several reas jns. First, the 
laboratory's vision of science-rich resources and schools working together v as grander than the 
mission of the science-rich resources, which was to reach out to teachers and students to give them 
access to the resources. The vision behind this mission did not include the two organizations working 
together to determine what schools needed, what the science resources could provide, and how best to 
use the resources to meet the needs. Further, outreach activities of any kind do not appear to have 
been high priorities for the museums and science centers. 

The design of the R&D Cadre calls for collaboration among the participants in research on 
topics of interest to all of the entities in the region. The fact that several of the entities have chosen 
not to participate in two of the studies has caused some tension in the cadre. One of the wealthier 
entities in the region declined to permit data on its school facilities and finances to be included in the 
cadre's first report. Laboratory staff maintain that the entity did not want information describing its 
relative wealth made public to the entity's poorer neighbors, A representative from the entity insisted 
that the lack uf participation was because a Qcw chief auiuUiisUaiivc officer concluded thai enough 
data were already available and that it would be unwise to commit resources to gather new data. Two 
other entities-also wealthy-bave declined to participate in the cadre's study of equal access. 
Laboratory staff report that they have worked hard to overcome these difficulties and to encourage 
full participation. 

Laboratory staff who work on the Families First Project are helping the staff in the social 
service agency develop a case management approach to working with families. At the heart of this 
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strategy is careful diagnosis of family needs followed by the development of a plan for intervention 
that draws on the services of a number of different agencies. Success depends in part on the case 
managers* knowledge of local resources and (he ability to marshal them as needed. A laboratory staff 
member conducts regular case meetings for individual families that arc served by multiple agencies. 
These meetings are occasions to assess the families' progress, to coordinate the services of the various 
agencies, and to model the case management strategy for agency staff. In this activity, collaboration 
is a central element in the content of the assistance, although as the laboratory staffs experience 
indicates, convening the various agencies for these sessions and getting them to work together on 
other tasks is not without complications. One laboratory staff member summarized the lessons as 
follows: 

We have learned several things from our work in Marin City. First, we are still in the 
wilderness as far as making collaborations work when funds are low. The issue of agency 
survival has taken center stage. Second, I've been very frustrated by the effects of categorical 
funding. The federal government, private foundations, and other agencies all think in terms 
of categories. We need interagency reform. Third, I'm not sure whether our society is 
committed to solving the problems or just having the illusion that we're solving the problems. 

Building intra-agency collaboration at the state level as well as collaborations between state 
and local education organization posed formidable challenges to laboratory staff who worked on the 
Missouri School Improvement Program. Once the basic framework for the new accreditation system 
had been developed, it was necessary to muster both support and a willingness to participate in the 
program. At the state level, this meant drawing together staff from throughout the state department 
of education to review and approve the new system and to join in the review process. In addition, it 
meant persuading school district administrators, teachers, and principals that they, too, should join the 
state department as partners in the process. The primary mechanism for building the partnership 
around the Missouri School Improvement Program was a 100-member statewide advisory committee 
that included state department staff, district administrators, principals, teachers, and school board 
members. The committee met two or three times a year for several years to review progress, to 
provide feedback on the design, and to build political support for the program. State department staff 
and laboratory staff worked together as facilitators for these meetings, and the laboratory provided 
significant logistical support, including funds for travel and lodging. 

The laboratories' decision to take on the issue of collaboration in these activities added a large 
measure of complexity to their work and has resulted in projects and tasks that are quite ambitious. 
In two of the four activities. Families First and the Missouri School Improvement Program, attention 
to c Maboration accounted for a large portion of the investment in the assistance activities, and there 
is evidence that the investment paid off. The R&D Cadre also demonstrates progress in this area. 
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although ihe investment of project resources is proportionately much less than in the first two. The 
laboratory's investment in Strengthening Science Partnerships has yielded few payoffs. In our view, 
the lahoratory should not be faulted for its vision of what is needed. The laboratory does, however, 
appear to have misjudged the complexity of the task of building these new partnerships. 

The Limits of Technical Assistance for Capacity Building 

In these activities, the laboratories set out to work either with particular organizations (e.g. , 
museums and science centers, a school district's Chapter 1 schoolwide projects office, a social service 
agency) or with a group of individuals (e.g., the R&D cadre and staff from a state department of 
education). In most cases, the choices also reflected laboratory programs' interests in addressing 
particular issues or problems-increasing the accessibility of science-rich resources, improving social 
service delivery systems, developing a data-based understanding of regional needs, and improving the 
administration of Chapter 1 programs. However, once these choices are made, the factor that most 
influences what the laboratory provides in the way of assistance is the agenda— or the absence of an 
agenda— of the organization or individuals the laboratory assists. Thus, assistance in developing the 
Missouri School Improvement Program was shaped by the goal of replacing an input-based 
accreditation system with one that was more equitable and explicitly linked to school improvement. 
Assistance to the Philadelphia Schoolwide Projects Office was shaped by the district staffs interest in 
developing their dissemination and technical assistance capacities. What is important about these two 
examples, as well as the other activities reviewed here, is that these agendas were not particularly 
well defined when the laboratories begin the relationships, and they do not appear to have been 
particularly high priorities in any of the organizations or for any of the individuals that received 
laboratory services and resources. 

In the early phases of these activities, the absence of commitment to clear agendas or tasks 
meant that laboratory staff had to work hard to generate interest and identify specific opportunities for 
assistance. As a member of the laboratory team that works on Strengthening Science Partnerships 
told us: 

We must convince museums that teacher training is important. Teachers are often not getting 
(training) and as a result they fall back on textbooks... .We need to build a capacity tor 
teachers and scientists to talk to together.... There is a communications gap. Scientists often 
just spew facts. Overall, we would like increase museum's understanding of the for training 
and substance. Just fun is not enough. 
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Despite several promising starts on long-term relationships to accomplish this objective, 
laboratory staff found their issue to be a hard sell with most museum staff. In general, these 
organizations appear to have relatively limited views of outreach and training, and building long-term 
relationships with schools and school districts seems to be a relatively low priority. During the past 
several years, laboratory staff have been fairly aggressive in marketing their services and their vision 
by offering short-term help on specific tasks. For example, laboratory staff have evaluated several 
museum programs and instructional packages as a way of "getting a foot in the door" and of 
beginning to convey their goal to the museum staff. A concluding observation in an evaluation report 
to a museum illustrates the laboratory's strategy: 

The [City] Children's Museum has an interest in connecting its science efforts with at-risk 
students and their teachers through providing effective teacher materials and kits. The 
evaluation of [program] kits links many of these interests and also enables a new interest born 
of the synergy of partnerships: exploration, the setting for science inquiry that science centers 
provide and science educators in schools seek to provide. By allowing a look at how children 
explore and how outreach materials can convey a spiru of exploration to teachers, this 
evaluation begins an examination of that substantial partnership issue. 

The laboratory was more committed to this partnership than either the museum or the schools. 
The evaluation, which involved classroom observations and interviews with teachers, was a service to 
the museum. The museum did not participate in the process, nor did we find evidence to suggest that 
the museum used the results. This pattern was fairly common in the laboratory's attempts to build 
relationships with the science-rich resources. As one somewhat frustrated member of the staff 
pointedly told us: "We're like an icicle. We can make [an impact] when we come down, but 
eventually we melt away." 

By way of comparison to this experience, laboratory staff working with the Philadelphia 
Public Schools were able to build on their work on a small task-four case studies of Chapter 1 
projects for schools to use in program improvement efforts— to establish a working relationship with 
the district, and, more important, to begin to help district Chapter 1 staff appreciate the need for 
dissemination and technical assistance as components of school improvement. Following this initial 
task, the laboratory conducted additional case studies of individual schools and provided a variety of 
other assistance to the Schoolwide Projects Office. 

In both of these cases the recipients of laboratory services were glad to get the services, but 
neither the services nor the long-range goals the laboratories intended to achieve were priorities for 
the recipients or for the organizations in which they worked. 
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Laboratory assistance in the Missouri School Improvement Program is a third example of 
laboratory staff trying to work with recipients to identify long term goals and the kinds of assistance 
necessary to achieve them. In this case, the laboratory was providing training to state supervisors, 
who, among other things, were responsible for the state accreditation system. Through a prolonged 
series of conversations, a laboratory staff member and a member of the staff of the state department 
of education gradually evolved planning and strategy sessions concerning the role of the supervisors 
and the problems with the accreditation system. These conversations led to a decision to try to design 
and implement a new state accreditation system, and laboratory assistance was intended to contribute 
to this process. 

The fundamental difference between the first two examples and the third one is that in the 
Missouri School Improvement Program, the capacity building goals and an agenda for technical 
assistance emerged from interactions between the laboratory and participants. The laboratory did not 
bring a goal or an assistance agenda to these interactions. The result was that the goal of the 
laboratory's assistance was, from the outset, important to those who were receiving assistance, and it 
became a priority of the state department of education. As one of the Missouri participants put it, 
"We started to think about what a new classification system would look like. Gradually, we began to 
evolve a series of standards, and we talked about what to collect from school districts. In time and 
after a lot of conversations, we became convinced that we had a good idea." 

What It Takes 

Our findings about the lack of commitment to these activities and the absence of basic 
capacity to become engaged with them c*.j not lead us to conclude that these activities were bad ideas. 
These findings do, however, suggest iriportant implications of the choices that the laboratories have 
made. For example, a consequence of choosing to work on issues that they have defined and that 
they think are important is that the laboratories have had to devote time and energy to, persuading the 
participants that these issues are important to them. Similarly, organizations and individuals with 
limited capacity require more basic kinds of assistance than those that are already working with some 
degree of success. These choices stand in contrast to decisions to work with organizations on 
activities to which they have made commitments and for which they have some capacity. The former 
choices carry greater risks and require Very different starting points. Given the starting points, these 
activities require long periods of time and more resources before there is a chance for even modest 
impact. 
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At the beginning of these activities, none of the participants or the organizations in which they 
worked had any commitment to working with the laboratories to buiid their capacity to improve 
schools or school districts or, in the case of Families First, to improve the quality of services to 
families. There is no evidence that they were actively opposed to the idea. Instead, they had just not 
considered it. This meant that the laboratories spent a considerable amount of time marketing 
themselves as partners, identifying development goals, and negotiating specific assistance tasks. 
When this process worked, as it did in the Missouri School Improvement Program, the partnerships 
developed and there was progress in achieving the development goals. When the process did not 
work, as in the case of Strengthening Science Partnerships, a laboratory may go on providing one- 
time or shorvterm assistance to many organizations as it trolls for partners. In between these two 
situations, a laboratory may continue providing assistance without ever really becoming a partner or 
contributing much to the capacity of individuals or of the organizations in which they work, as in the 
case of RBS's support to the Philadelphia Schoolwide Projects Office. 

Building commitment to long-term goals as well as shorter-term assistance activities was 
hampered by the relative absence of institutional capacity in all of the partner organizations. This 
manifested itself in several ways. First, and most obvious, the partners had few, if any, resources to 
devote to the activities. Therefore, it was up to the laboratories to provide the resources. A 
laboratory pays for the meetings of the R&D Cadre, although the entities contribute the participants' 
time as in-kind support. Another laboratory provided a significant amount of training to support 
implementation of the Missouri School improvement Program and, as we repotted above, paid for 
many of the planning and consensus building activities. The museums and science centers with which 
a third laboratory attempted to work had almost no resources to devote to professional development 
activities, and often there was only a single staff member assigned to this component of services. 

Second, there was limited technical capacity on which to build. The entities in the Pacific 
region have almost no experience in research. Museum and science center staff were unfamiliar with 
the fundamentals of professional development and program evaluation. Staff in the Philadelphia 
Schoolwide Projects Office had not provided much technical assistance to schools and were 
inexperienced in disseminating information. The good news is that the limited capacity created 
numerous options for laboratory assistance. The bad news is that it made it difficult to even 
conceptualize problems. 

These problems are compounded when laboratory staff work with only one or two people in 
an organization, particularly when these individuals do not have much authority. Members of the 
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R&D Cadre can deliver a report to the chief administrative officers of the education systems in 
entities, hut they have little or no influence over the allocation of resources for research or for 
subsequent development of the capacity to conduct research. Museum outreach activities appear to be 
peripheral to museum activities, and the staff were often assigned to these activities on a part-time 
basis. The Missouri School Improvement Program illustrates what can happen when participants have 
a degree of authority and influence in their organization . In this case, one of the two key participants 
had some responsibility for the accreditation system as well as access to state-level policymakers. 
Initially, he was able to use his influence to call attention to the emerging plans. Later, with 
considerable assistance from the laboratory, he used his influence to build support and consensus 
around the new system. 

Time 

As our findings about all of these activities indicate, time is an important ingredient in this 
technical assistance. Laboratory staff usually devote a lot of time to these activities and, in most 
cases, they do so over a period of at least several years. Laboratory assistance in designing the 
Missouri School Improvement Program spanned more than five years, and four more years were 
required to complete the first phase of implementation. Laboratory assistance to the R&D Cadre and 
RBS's assistance in Philadelphia began with the current contract period and have continued without 
interruption. Finally, in addition to whatever routine contacts there are, laboratory staff are available 
when they are needed. This has symbolic as well as strategic importance. Through time, laboratory 
staff and, to a lesser extent, the laboratories as organizations come to be seen as reliable partners. 
"They are there for the long haul, and they can help us put out the fires." 

Resources 

In addition to having time to work with the recipients of technical assistance, the laboratories 
have had some success at marshalling other resources, especially money to support the various 
activities. The Missouri School Improvement Program is the best example of a laboratory making 
resources available as they were needed. This was due, in part, to the fact that the person who led 
this activity was also a senior member of the laboratory staff. Resources were available from the 
laboratory program area that housed the activity as well as from several other program areas. People 
in Missouri who are familiar with the laboratory's efforts frequently mentioned the importance of the 
resources and the fact thai they appeared to be available quickly and easily. Thus, one of the keys to 
effective allocation of resources is flexibility. 
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In terms of the overall portfolios of laboratories' activities, these activities are relatively small 
investments. From another perspective, they are rather large investments. This assistance represents 
a significant concentration of staff lime-often by senior staff-on services to very few people over 
two, three, or even four years. When an activity takes off, the costs escalate accordingly. Ten years 
of assistance to the Missouri School Improvement Program represented a very large expenditure 
within the laboratory's portfolio. If more science-rich resources had responded to laboratory 
overtures to help develop their outreach and professional development capacities, the laboratory would 
have faced potentially difficult choices about its investment; In the end, it is accurate to describe 
these activities as small but not inexpensive. 
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V. NEUTRAL GROUND FOR CONVENING 



We did not set out to examine the role of laboratories in convening regional groups. As we 
investigated other activities, it often popped up on its own as an important and unique niche for the 
laboratories in the worlds of education policy and practice. Two of the activities we briefly describe 
below had originally been selected for in-depth study as examples of innovative technical assistance; 
the others we came to know as participants brought them to our attention during other discussions. 

Convening Activities 

Among the convening activities that came to our attention are three that represent ongoing and 
significant laboratory investment. 

SERVE-Une 

SERVE-Line is an on-line information system that enables teachers, administrators, and media 
specialists to request information about educational issues from laboratory staff and to communicate 
with other educators in the region. SERVE staff as well as staff from other laboratories also use the 
service, which offers the users many options: an information request service complete with ERIC 
searches, electronic mail, shareware, online curriculum guides such as CNN Newsroom, discussion 
forums and bulletin boards, calendars of events, a database of laboratory publications and products, 
and education news items. A participant must have access to a computer, phone line, and modem to 
use the service, which is reached through a toll-free number. A school pays a $25 yearly fee to use 
the service including the bulletin board and chat options; without those options it is free. 

The information request service, which operates out of SERVE'S Florida base, fills on-line 
requests by U.S. Mail. Participants use e-mail to network with educators across the country; 
shareware enables them to preview and download software. SERVE-Line started running in August 
1991. It currently employs 3.5 full-time equivalents (PTEs) to run the service, which uses 16 phone 
lines. About 14,000 people in the region have signed up so far. 

The teachers we interviewed were enthusiastic about SERVE-Line and praised it as a user- 
friendly, multi-purpose tool. White several teachers spoke about information searches, e-mail was 
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mentioned by nearly everyone; teachers use it to communicate with other teachers that they know* to 
jump in on conversations on topics of interest, or to "convene. " One teacher in Alabama* for 
example, uses SERVE-Line to keep up with Education Week articles, to poll other teachers in the 
states for input on current issues, and to meet electronically with other members of the Leadership 
Council of the State Teachers' Forum. 



pr el's Annual Pacific Educational Conference 

For the past 1 1 years, PREL (or its predecessor unit within NWREL) has co-sponsored, along 
with a host entity, a three-day conference for teachers, administrators, and other interested educators 
in the Pacific region. For more than a thousand participants, it is a time for sharing problems, 
building skills through workshops, and making friends. It is also a very significant time for the 
islands to develop and sustain a sense of regional unity despite differences in language and culture. 

The conference has evolved over the years from one in which participants came to hear news 
from the mainland to one in which they share expertise and ideas. During the first few conferences, 
all presenters were from the mainland; recently, on the other twnd, between 85 and 95 percent of the 
presentations have been given by Pacific natives. 

The co-sponsoring entity benefits from this collaboration in several ways. Its teachers get the 
opportunity to attend and meet colleagues from around the Pacific, and this may be the only 
professional event of their lives. Conference planning-which usually takes a year-brings a lot of 
attention to education on the host island among islanders and across levels of government. This in 
turn may result in substantial capital improvements for the local schools: the PA system or the air 
conditioning get fixed, rooms are painted, and grounds are cleaned up in preparation for distinguished 
visitors. 

SEPL's Policymaker Network 

As part of its State Policy and Planning Service, SEDL has developed a regional network of 
state policymakers. The network evolved from SEDL's Ed-Aide service, which responds to the 
informational needs of regional and state policymakers with tailored "rapid response 14 packets 
designed to inform educational policy decisions. From the contacts cultivated through Ed-Aide, 
SEDL realized that it was in a unique position to know and understand the policy issues common to 
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many of it* information recipients. Laboratory staff then decided to develop a process tor convening 
key policymakers in the region. 

The policy network includes 35 to 40 key policymakers across the SEDL region for 
participation in the periodic teleconferences and two *networkshops H each year. The teleconferences 
are designed to identify common concerns and issues that can be the focus of the biannual 
networkshops. At the networkshops, SEDL brings in national and regional experts to share insights 
and to facilitate discussion among the participants, SEDL typically publishes an issue paper and 
distributes it throughout the region. Network participants give the meetings high marks for quality, 
organization, and usefulness. Outside the network meetings, SEDL acts as an intermediary for 
members. Eventually, SEDL staff would like to see policymakers initiate direct contact among 
themselves. 

Other Convening Activities 

We were told about a number of shorter or more focused events that were popular among the 
laboratories' constituencies. For example: 

• NCREL helped people at the Center for Education and Work organize a policy 
institute for the Great Lakes states. As a co-sponsor, the laboratory helped make 
logistical arrangements, got the chiefs together along with teams of 2-8 people from 
each state, and provided three staff members as facilitators. 

• RBS helped Maryland state department staff put together a task force on student 
discipline and motivation (the Task Force on Recognition of the Academic 
Achievement of Students), providing materials and attending monthly meetings. The 
laboratory used a team approach to facilitate rethinking of Pennsylvania's Chapter 1 
program, including organizing a three-day institute together with the regional 
Technical Assistance Center, RBS also helped the Pennsylvania state department put 
together a collaborative early childhood initiative that included staff from Head Stan. 
Chapter 1. child care organizations, and school districts. 

• FWL's Policy Support Program convenes its Policy Advisory Group of regional 
policymakers to address common issues and present concerns. The laboratory 
recently held a conference on systemic reform called "Rethinking Time." The 
laboratory also leads roundtables of superintendents from the 8-10 largest districts in 
California who determine the agenda that best fits their needs; the latest meeting 
addressed implementation of Goals 2000 grants. 
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Our interviews about other laboratory activities suggested that some of the participants 1 
favorite features of more elaborate efforts are the times when they get together with their colleagues 
to plan or design some aspect of the project. For example, teachers we spoke with in the AEL Study 
Groups-where teachers collectively research and report on a particular topic-referred often to the 
gathering feature; otie said, "The networking is the best part about this whole thing. . . . Those 
teachers (from another school] want to try what we did this year/ Similarly, representatives from 
NE/IS's Teacher Induction Working Group praised the networking opportunities at meetings 
facilitated by the laboratory once a month. (This project is launching a Regional Mentoring Network 
that reflects a similar process). 

Also at the NE/1S Laboratory, the Partners activity within the Designing Schools initiative 
relies on gatherings that constitute an elaborate network with varying levels of involvement. Partners 
are a loose confederation of schools that die invited to participate in an annual conference, send 
representatives to meetings of a steering committee, send representatives to thematic "Working 
Parties/ and communicate through computer networks. 

An important objective of SEDL's Leadership for Cnange is to create networking 
opportunities for the educators who attend. A laboratory staff member said that the participants often 
say, "The only time we get together is when SEDL calls a meeting." A number of the activities at 
the training session we attended are designed specifically to encourage participants to share 
information and to establish contacts with each other. 

Convening as a Category of Laboratory Activities 

As noted above, the laboratory convening function was not the subject of our initial analysis, 
but rather emerged as an important issue during the course of our other data collection. Because we 
did not probe at all for reactions to this role, we assume that it has particular importance for 
participants in laboratory activities who brought it up. Many respondents offered the perspective that 
laboratories are particularly well-suited for drawing people together in productive ways. There are a 
number of reasons that make this niche a logical one. 

The laboratories are trusted as neutral players* This was very clear in our interviews with 
policymakers about the short-turnaround information requests: recipients are assured that they will 
not receive a party line. The combination of this lack of bias with broad background knowledge is 
seen as invaluable by educators. Because the laboratories are not major stakeholders in key policy 
issues, they are especially credible resources. 
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The laboratories have an enormous range of contacts that cross stale and disciplinary lines. 
They are in a unique position to draw on expertise across governmental levels as well as from school 
districts and universities. This can result in new networks of people who have a common interest hut 
who are normally separated by traditional turf concerns or simply isolation. 

The laboratories have the flexibility to sponsor gatherings on an ad hoc basis. During 
the course of their regular work, laboratory staff frequently become aware of particularly salient 
issues that recur among their colleagues and participants. Many state policymakers are concerned 
with the implications of systemic reform and Goals 2000; others are looking for ways to encourage 
new types of professional development. When laboratory staff hear these concerns voiced repeatedly, 
they are able to gather interested parties together on relatively short notice-with an agenda that is 
tailored to expressed needs. 

Results 

Again, the lack of systematic data collection for this category makes it impossible to 
determine the effects of networking activities with any confidence. In any case, the impact of such 
events is likely to be difficult to pin down, as with the short-term information events discussed in the 
previous chapter. While behavior change is relatively rare, there is real potential for guidance on 
policy options and broader awareness of needs and possible solutions to problems. 

From the perspectives of the participants who happened to mention these gatherings, the 
convening function is a resounding success. There is obvious self-selection here; we would be far 
less likely to hear spontaneous descriptions of events that were less than memorable. With that in 
mind, it still seems that laboratories' convening activities accomplish several goals. There are a 
number of reasons why this makes a great deal of sense. The logic may be slightly different 
depending on the role groups involved. 

Participants Are Delighted with the Events 

Teachers and school building administrators are notorious for being relatively secluded 
professionals. They can benefit enormously from the intellectual stimulation that comes from the 
opportunity to meet new colleagues. In this way, the laboratory contributes to a sense of 
professionalism that is sorely lacking for most educators, even as it is understood to be increasingly 
essential for improving schools. 
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The value of *al lowing school people the time to work together on common tasks as part of a 
larger entity--* profession-should not be underestimated* When other professionals attend 
conferences, dramatic behavior change does not result, but they typically come away from the events 
with a renewed sense of purpose, some different perspectives on innovations in the field, and possibly 
energy and enthusiasm. Teachers and administrators are no exception here, and the laboratories 
provide a real service when they furnish teachers with the time and the context for this growth. 

The Laboratories Create a Context Conducive to Collective Problem Solving 

Policymakers and higher level administrators describe the effects somewhat differently from 
teachers, although in an equally positive light. For this group, convening events are seen more 
specifically as opportunities to solve common problems through an exchange of ideas* The diversity 
of the cast of characters-colleagues whom they do not normally encounter in their daily work lives, 
as well as different role groups— may make for richer solutions to problems. 

An important aspect of laboratory-sponsored activities for policymakers is the protected space 
to study and reflect. Both time and space that are no one person's territory are scarce resources* At 
FWL, the written objectives for the urban superintendents' meetings include the opportunity to discuss 
common interests in a "informal and private, rather than public* environment. Within this setting, 
laboratory staff hope that 'discussion topics will emanate from superintendents 9 interests in 
instructional improvement, as opposed to their political exigencies. " In this unusually sheltered 
environment, constructive dialogue about possible responses to legislation and mandates is much more 
likely to occur, 

Laboratory-Sponsored Events Can Spawn Further Collaboration and Expanded Networks 

When the AEL Study Groups get together, for example, they are offered the possibility of 
assuming new roles as researchers that may prompt them to organize similar efforts within their 
schools, districts, or local associations. The Teacher Induction Working Group at NE/IS laboratory 
has in turn generated the much larger Regional Mentoring Network. PREL staff members arc proud 
of the lasting professional connections that emerge from the annual conference. This year, for 
example, an association of school board members from the various entities was formed and met to 
discuss common concerns. 
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Convening Events Increase the Laboratories' Engagement with a Range of Educators 



In this sense, laboratory-spoasored conferences and meetings make important contributions to 
the laboratories* ongoing dialogue with the field. In facilitating these sessions, laboratory staff learn 
first hand about the most burning issues and regional needs. Similarly, practitioners and 
policymakers alike are exposed to the laboratory as a potential resource to draw on in the future. 

Limitations 

The Value of Convening Activities Mav Not Be Readily Understood bv the Public 

In an era of sharp budget cuts in education, all professional development activities run the risk 
of being understood as "frills/ ami as such are often the first to be eliminated. Furthermore, these 
events may not be appreciated by OERI unless they are designed to convey information about the 
federal agenda. For example, the importance of the PREL conference as a central ingredient of the 
laboratory's regional work had to be argued quite vigorously when contracts were awarded. In short, 
the current incentive structure might not be the best for encouraging these ventures. 

Effects Are Difficult to Measure 

Broadening perspectives and enhancing professionalism are not changes amenable to 
traditional assessment. This may partially account for the tenuous reputation described above. Of all 
the things laboratories do, convening role groups may be seen by critics as the most removed from 
potential classroom impact. 

Because of High Client Satisfaction, Laboratories Mav Run the Risk of "Indiscriminate" 
Convening 

As with workshops, hosting successful conferences is highly reinforcing. Enthusiastic, 
committed participants are appreciative of hard work and a job well done, which makes it tempting to 
do often. The more comprehensive, systemic work that the laboratory undertakes may be much seem 
much less rewarding in the short run, but is crucial to the laboratory mission. 
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VI. CONCLUSIONS AND POLICY IMPLICATIONS 



Although the work of regional educational laboratories is hard to characterize in a simple way 
because the activities vary sc much in purpose, content, duration, and type of participants, we can 
nevertheless draw some broad-brush conclusions across most of the activities we studied. Our aim 
here is to identify and analyze commonalities in the overall laboratory program. With the caveat that 
not every conclusion applies to every activity we studied, we still believe the following strengths and 
weaknesses are sufficiently widespread in the program to provide a reasonable basis for policy 
choices. 

Cross-Cutting Strengths 

Participants in laboratory activities almost uniformly express satisfaction with the experience. 
Moreover, the presence of specific, common themes in their reports suggests that there are in fact 
distinctive strengths in the work of laboratories (whereas more global, vague praise would have 
suggested to us that participants were showing more politeness than discernment). 

Participants Sa v the Activities Are Useful and of High Quality 

One of the strongest tests of the work of laboratories is in the extent to which participants 
report new behaviors as a result of this work. Such reports are common among participants in 
development activities, many of whom say they are applying the new skills that the laboratory 
intended to impart. These skills are often techniques of classroom instruction, and they sometimes 
include organizational actions such as goal setting, planning, and increased professional discussion 
among teachers. For these participants, the combination of products and processes with technical 
assistance adds up to a powerful intervention. 

To our surprise, we also found that almost 40 percent of the participants in workshops or 
recipients of products reported a change in behavior-usually a trial of "something new" on the job-as 
a result of the information provided by the laboratory. The fact that these trials may lack much depth 
or permanence is balanced by the fact that they represent a fairly widespread impact from a relatively 
low-cost laboratory activity. Three-quarters of the survey respondents who attended workshops or 
received a product also reported passing laboratory materials along to colleagues in their 
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organizations. Similarly, the information transmitted to policymakers or their advisors has often been 
put to work, according to recipients. In a different category of activities, technical assistance to build 
capacity, the participants spoke highly of the usefulness of ihe laboratory's help, and in some cases 
they reported continuing application of what they h*ve learned from that help (although the lasting 
effects have fallen short of the original intentions in some cases within this category). 

"Quality " is a term with many meanings. Rather than try to define or judge quality 
ourselves, we asked participants for their assessments of the quality of specific examples of laboratory 
work, and most rated it as high. In the survey of workshop participants and product recipients, these 
events and products were generally reported to be of higher quality than comparable offerings from 
other sources with respect to content and presentation. Recipients of policy information also 
compared the laboratories favorably with other sources of similar information, including universities 
and national associations. Convening activities became a subject of description and analysis in this 
study on the strength of participants' reports, which often noted the exceptional quality of the overall 
experience and credited the laboratories with skilled preparation. The participants in development 
projects and in technical assistance for capacity building, whom we interviewed in person or by 
telephone, were often effusive in praising the quality of the laboratories' work. We elaborate below 
on specific strengths attributed to the laboratories* work, which reflect dimensions of quality that we 
identify in the reports of participants. 

Laboratories Are Credible Sources of Help 

The reported credibility of information, products, and assistance from laboratories fulfills a 
policy expectation for publicly supported research, development, and assistance. A key rationale for 
a public investment in these activities is that it enables organizations to amass research-based expertise 
and to apply this expertise impartially, without the distortions that could be introduced by commercial 
self-interest. And, in fact, a number of participants explicitly recognized this as a strength of the 
laboratories' work. Laboratories can be trusted to give an honest reading of the evidence on policy 
issues, according to recipients of their policy-oriented syntheses. Many participants in development 
efforts observed with surprise that the laboratory staff were willing to revise their products and 
processes rather than trying to sell them in their existing form. Technical assistance in both 
development and capacity-building activities is reportedly distinguished by a genuine openness to 
understanding the participant's situation in depth and tailoring the help accordingly, rather than force- 
fitting a particular solution to a problem. 
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An important ingredient in credibility is the disciplined inquiry that underlies laboratory work. 
Participants referred to the research base behind laboratory products, processes, and events as a 
source of credibility. Some said that they themselves found the existence of research important and 
persuasive; others said it helped them make a case to decisionmakers (e.g., a school board or a 
committee of the legislature). 

Laboratories Make Long -Term Commitments to Topics and Partners 

Even though the laboratories are funded on a. five-year cycle, many of the activities we 
studied have longer histories (and some have roots in individual professional histories that go back for 
decades). We think that there might have been room for acceleration in some of the longest 
development activities, but we are still inclined to agree with laboratory staff that the gradual 
maturing of an activity can permit a rich and eclectic mixture of research, practice, and evaluation to 
inform the products and processes developed. Some laboratory development efforts are intellectually 
ambitious, and realizing these ambitions probably does take time-elapsed time as well as intensive 
commitment— on the part of the development team. 

The fact that laboratories support long-term programs of research, development, and 
assistance in particular areas can help strengthen even those events and products that represent limited 
time commitments for participants and recipients. Although we do not i now which specific features 
of these events and products participants value the most, it is plausible that the perceived quality of 
wurkshpp content and other laboratory products derives in part from the depth and scope of the 
related program of work at the laboratory. Other technical -assistance providers, not funded to carry 
out programs of R&D, may have difficulty matching the depth of knowledge brought to bear in the 
most highly regarded laboratory presentations. 

Long-term commitments to partners represent another dimension of strength in the 
laboratories" work. Particularly in our category of technical assistance for capacity building, 
participants attributed successes to the laboratories' perseverance in the face of delays and setbacks. 
The repeated interactions with participants in development efforts also contributed to the effectiveness 
of these activities, according to participants. 
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Laboratories Are Boundary Spanners 

In different ways, most of the activities studied here represent a synthesis of some kind. Must 
activities hlend research knowledge with insights from practice; some bring together different strands 
of research; several bring a policy sense to issues of practice or vice versa. This capacity for 
synthesis strengthens the laboratories as a resource, according to our respondents. Many spoke highly 
of the practicality of laboratories' ideas, contrasting them with more theoretical (and, in the case of 
policy, more partisan) formulations. Practitioners who participated in development efforts told us that 
they were initially surprised to be treated as fellow professionals by the laboratory staff, then went on 
to describe how much they learned from the effort. The incorporation of practice into R&D, in other 
words, goes well beyond mere courtesy and creates real learning opportunities. In another category 
of activities, the capacity to span conventional boundaries clearly contributes to the perceived value of 
events convened by laboratories, which participants often described as unique in their freedom from 
turf issues. 

Cross-Cutting Weaknesses 4 

Participants in laboratory work offered relatively few critical comments about the experience, 
but the criticisms provide some insight into weaknesses of the laboratories' work. In addition, our 
analysis of data on the operations and effects of these activities enabled us to spot missed 
opportunities that might not be so apparent to those more closely involved in the activities. The 
weaknesses described here, then, reflect a mixture of participants* reports and our analyses. 

Development and Assistance Could Reflect More Engagement with the Field 

Some of the products, processes, and assistance efforts proffered by laboratories reflect 
optimistic assumptions about the preferences or agendas of practitioners-in other words, they reflect 
failures of marketing. The public sector sometimes recoils from the idea of marketing, equating it 
with selling or the artificial creation of consumer wants, but in fact effective marketing brings an 
understanding of real-world needs, wants, and interests into the development and refinement of 
products anti services (Kotler, 1991). We have seen flawed marketing in the laboratories' 
development and attempted dissemination of many large, unwieldy compendia of research findings; in 
some efforts to enlist educators as volunteer disseminators of laboratory processes and products; in a 
few policymakers' perception that some laboratories are only willing to do work that advances a 
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particular agenda; and in the assumption that particular organizations strategically situated lo assist 
schools actually share a laboratory's agenda of school improvement 

The remedy that we would suggest builds on the record of successful engagement with the 
field found in many development and assistance efforts. We urge that laboratory staff spend even 
more time learning about the actual capacities, agendas, felt needs, and latent needs in schools and 
other agencies, so that the design of development and assistance efforts is continually informed by 
these realities. Conventional needs assessment, which is often a compilation of lists of high-priority 
topics, does not have the necessary depth and does not take enough account of the dimension of 
capacity. Field testing, on the other hand, represents a tremendous learning opportunity that should 
be approached in a spirit of open inquiry. This does not mean that laboratories should simply wait 
for practitioners and policymakers to tell them what to do-on the contrary, they must bring an 
informed and critical imagination to the interaction-but they must watch and listen. 

Laboratory activities that fall outside the conventional boundaries of development projects, 
such as workshops, are also opportunities for two-way engagement in the field. Sometimes the 
simple evaluation forms distributed to workshop participants or product recipients ask for information 
about other needs that the laboratory might address, but these are limited in two ways: they focus on 
needs (rather than capacities or preferences), and they are not dynamic exchanges. Instead, we 
suggest that laboratories make more formal efforts to learn from the conversations that take place 
during workshops or in focus-group trials of a product. These can be seen as the occasions for 
inexpensive probes into the field that can help inform related efforts in development and assistance. 

The Targeting of Products and Services Deserves More Attention 

Although we are genuinely impressed by the favorable comments that we gathered from 
participants in the laboratories' activities, we worry in some cases about the ratio between satisfied 
participants and dollars spent. Whether in a development activity that does not seem destined for 
second-generation dissemination at a reduced per-participant cost, or in technical assistance for 
capacity building that has not yet resulted in trickle-down benefits beyond the organization directly 
helped, laboratories are sometimes prone to delivering very good services to very few participants. 
Not every activity should serve a huge volume of participants, of course, but we think the laboratories 
could do more to press for efficiency in their work, especially by making a more concerted effort to 
solve the problems of "scaling up" in development efforts and by targeting capacity-building efforts to 
organizations that start with a strong base of commitment and capacity. Laboratories also should have 
the flexibility to invest heavily in their most promising activities so as to expand their reach. 
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In addition to this issue about the number of participants, there is an issue of who the 
participants are. Laboratories should make careful choices about targeting if they pursue our previous 
point about engagement with the field. A laboratory can see itself as having an ongoing conversation 
with its region, trying to impart knowledge where it will do some good and also trying to learn from 
partners who have a range of perspectives. For activities that involve a serious time commitment 
from participants, this requires the laboratory to strike a balance between addressing acute needs and 
choosing partner organizations that bring commitment and capacity to the working relationship. It 
might also require attending to different dimensions of capacity. (Organizations, like students, might 
have multiple "intelligences. 11 ) For short-term activities, laboratories should work hard to choose an 
appropriate balance between events that are closely tailored to participants' interests and those that 
offer a few new ideas to the more casual participant. 

In the relatively few cases where we found disappointment in a working relationship (or a 
short encounter) with a laboratory, the reason was often that the parties brought different expectations 
to the activity. At least for long-term activities, more time should probably be invested in finding 
interested participants or tailoring the involvement to the participant's interest and commitment. 

Laboratories Should Rigorously Scrutinize What Isn't Working 

Some of the field tests and other reviews brought to bear on laboratory activities embody 
systematic designs, formal documentation that captures findings, and commitment to use the 
conclusions -but most do not. There is a continual temptation to seek good news and favorable 
ratings; designs do not always capture a range of important effects and issues; and developers or 
assistance providers sometimes forge ahead in the face of what should be clear signals to reconsider a 
venture. 

Unless laboratories are being excessively cautious, some activities will turn out to be real 
failures, and almost all will have serious flaws in their early iterations. Thus, field testing and other 
occasions for reviewing the design and progress of laboratory activities should not be considered 
threats to be defended against or hurdles to be overcome. They are opportunities for posing a range 
of choices, asking hard questions, and seeing new possibilities. Laboratory managers and OER1 
should encourage evaluators and program staff to scrutinize the assumptions behind activities as well 
as the mechanics of execution and to radically change or abandon whatever is not working effectively. 
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Communication Within and Across Laboratories Should Increase 



This evaluation has afforded us an unusual opportunity to look across seemingly diverse 
laboratory activities, tease out their commonalities, and identify lessons for the future. It is an 
exercise that we recommend for the laboratories themselves. Although there is never enough time in 
the work day for much systematic examination of what has been learned in a project, we think that 
future laboratory activities would benefit from more cross-project and cross-laboratory discussion of 
lessons learned. Many of the professionals in laboratories are so engrossed in their own projects that 
they know little about their colleagues' work. This has two unfortunate effects: they cannot inform 
practitioners about the range of resources available from their own organization; and organizational 
learning is impeded. We have observed some efforts to decrease the isolation within and across 
laboratories, and we urge that these efforts continue and expand. 

Policy Implications 

The implications of these conclusions for OERJ are not always direct. The agency does not 
have hands-on responsibility for laboratories' work; it could mandate management practices, but 
organizations can easily distort or subvert these process controls. Instead, we believe OERf s 
opportunities are to be found in framing an RFP for laboratory work, in establishing performance 
measures and reporting requirements for the program, and in fostering networking and community 
* building within and across all the programs it funds. Most fundamentally-whUe maintaining 
responsible safeguards against worst-case problems such as incompetence or fraud-OERI can show 
leadership in framing and communicating a vision of the best possibilities in the work of laboratories. 

Contracted Activities 

Because the possible array of laboratory activities is so broad in content, participants, and 
design, government officials need a way to impart some structure to the RFP. The 1985 and 1990 
RFPs created categories of activities along functional lines, and one result in 1990 was the formal 
separation of technical assistance from applied R&D. In fact, as we have seen, this has had little or 
no practical effect on the work of laboratories; they meld the two functions in many large-scale 
activities. Thus, the new statement of work could continue to split these types of activities without 
doing much harm, as long as everyone recognizes that some of the best development builds on 
assistance and vice versa. But, on the whole, we would recommend encouraging laboratories to 
organize their work into substantive families of activities, in which development, some limited amount 
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of applied research, long-term and short-term assistance, and evaluation are organized to inform and 
support one another. 

The idea of families of activities would build on an existing strength of laboratory work-the 
spanning of boundaries (among research disciplines, between research and practice)-whi!e addressing 
the existing weakness that some isolated activities do not benefit from communication across projects 
or intense engagement with Che field. OERI could ask laboratories to demonstrate how sets of 
interrelated activities will be managed so that field experience is scrutinized through the lens of formal 
inquiry and vice versa. If a laboratory offers workshops (and our data suggest that the regions will 
want workshops), they should be tightly linked to longer-term development and assistance efforts for 
two reasons: workshop participants will benefit from the depth of knowledge brought to bear; and 
the laboratory's offerings in development and assistance will benefit from what is learned from 
workshop participants. (Note that this should not be labeled "needs assessment"; workshop 
participants bring capacities and ideas as well as needs, and these should figure prominently in the 
lessons learned by laboratory staff.) 

A different issue is also likely to arise in the specification of laboratory activities for the next 
contract period. Because laboratories are authorized under the Goals 2000: Educate America Act 
and are charged with some responsibilities under the Improving America's Schools Act, they may 
acquire an obligation to embrace a particular vision of reform content or process. We believe that 
OERI should strive for flexibility in this regard, bearing in mind that Goals 2000 is intended as a very 
broad umbrella for statewide-and, indeed, national-conversations about standards and reform 
directions. We advocate flexibility because harnessing the laboratories to a particular vision of 
reform, however solidly grounded in research that vision may seem, would seriously undercut the 
neutrality and hence the credibility that they now bring to policy discussions in their regions. A 
specific example can illustrate this point: if OERI tries to capitalize on the laboratories' success in 
convening regional conversations by charging them with convening lots of regional meetings as 
forums for communicating Administration policy, the success will be short-lived. 

Performance Measures and Reporting Requirements 

In cooperation with the laboratories, OERI has explored some options in the collection and 
reporting of performance measures for the program. The next five years will almost certainly see the 
application of some measures nationally along with the experimental use of other measures. 
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OERI policymakers know that counting participants in laboratory events provides only one 
view of the program's accomplishments. Although very low numbers may show a disturbing lack of 
energy or connection with the field, very high numbers may show superficiality in the program of 
work. It will clearly be important to develop measures of depth of engagement so that these can 
accompany measures of breadth of coverage. 

Our data indicate that "customer surveys" will result in high ratings for the work of 
laboratories. To the extent possible, it will be important to probe beyond measures of satisfaction 
with the form or content of laboratory services and products. In particular, it will be interesting and 
useful to ask participants how they use what they have learned from laboratories-and to ask them 
what, if anything, the laboratories seem to be learning from them. We would also advocate a real 
press for information on behavioral changes and, especially* on student effects that are plausibly 
attributable to the work of laboratories. Not every activity will have a plausible causal chain to 
student effects, but many will. Too often, measures of laboratory activity have focused on process 
rather than results. 

To combat the defensiveness now found in some evaluation and field testing in the laboratory 
program, OERI should send a clear message that it expects and will tolerate some reports of failure. 
This would be a change in the policy climate of the program, in which the laboratories have expected 
OERI to castigate them for missteps. Laboratories should be publicly praised for identifying their 
mistakes-provided that they show evidence that they are applying the lessons learned. 

Finally, the reports that laboratories submit to OERI should be much shorter and more user- 
friendly. This might or might not set an example that the developers of other laboratory products 
would emulate, but it could not hurt. 

Networking and Community Building 

We have observed that the laboratories' work would benefit from more opportunities to 
identify and apply the lessons that emerge from related work within and across organizations. OERI 
should require and facilitate communication among laboratories at the program level, especially 
among staff members who are working on substantively related projects. Laboratory evaluators 
should continue to meet as a group. The management meetings that now bring executive directors to 
Washington should also continue as forums for discussing overall program directions, but the 
executive directors do not have hands-on engagement with all of their organizations' work and thus 
cannot relay and reflect on a f l the practical lessons that are worth sharing across laboratories. 
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New legislation requires the laboratories to collaborate with other federal programs, including 
the institutes, the National Diffusion Network, and the comprehensive regional assistance centers. 
OERI should recognize that collaboration is time consuming and therefore expensive, especially when 
it is not very actively promoted, facilitated, and modeled by federal officials. A useful first step, 
then, would be for the laboratory team in OERI to build strong lines of communication with these 
other programs. This is now under way, and it can contribute to more effective communication in the 
field and, ultimately, to better service to participants. 



Vision 

Federal policymakers, if they have heard of regional educational laboratories, typically know 
little about the work of this program. Its amorphous charge and decentralized governance add to its 
ambiguity, from a Washington perspective, and a past history of disdain on the pait of some top 
policymakers does not help. In this evaluation we have tried to identify the major type of work that 
laboratories do, to highlight characteristic strengths and weaknesses in the design and execution of 
each type, and to Identify cross-cutting strengths and weaknesses within the range of activities we 
examined. Our aim has been to contribute to a program-wide clarification of what laboratories, in the 
current phrase, know and can do. Based in part on this information-and in communication with 
boards, managers, and professionals in the laboratory program-OEM should formulate a vision of 
high aspirations for this program, then hold the laboratories accountable for realizing and further 
strengthening the vision. This would differ from a minimalist policy of expecting success in 
conducting discrete projects, technical-assistance events, and regional meetings; it would require a 
greater degree of programmatic coherence and ambition from the government as well as from the 
laboratories. 
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APPENDIX 
Survey Procedures 



Survey Administration and Response Rate 

We worked with laboratory staff to identify workshops and similar activities suitable for 
study. Workshop and training activities studied included SEDL's "Country Stars: Promising 
Practices for Rural At-risk Students," RBS' "Urban Learner Framework," NE/IS's "The Making 
Change Game/ NWREL's "Classroom Assessment Training/ SERVE'S "School Safety and 
Emergency Preparedness: Reducing School Violence Training," FWL's "Issues in Teaching English 
Learners" and "Portfolios and Issues in Language, Culture, and Equity," and SEDL's "Leadership for 
Change/ We also surveyed recipients of McREL's product, The Systematic Identification and 
Articulation of Content Standards y^f bench marks . 

Laboratory staff or event organizers recommended by laboratory staff provided us with lists 
of potential survey respondents. According to those who provided the lists, all participants of an 
individual event were listed. We drew a systematic random sample of each list provided to us, with 
the exception of lists with fewer than SO people, where we sampled all participants. 

We administered surveys during April and May 1994, on a staggered schedule based on the 
availability of lists of potential respondents. In all, we sent surveys to 448 laboratory activity 
participants; 314 participants completed and returned the survey for an overall response rate of 70 
percent. The data analysis assigned weights to responses by activity to reflect the numbers of 
potential respondents. 

A copy of a survey is attached. Items were tailored to specific events and products, but this 
example includes the core content that was common to all the surveys. 

Follow-up Procedures 

We called all nonrespondents one to three weeks after mailing the surveys to encourage them 
to complete and return the survey as soon as possible. We mailed or foxed nother survey to 
potential respondents who had not received the survey or had misplaced it. When possible, we 
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conducted surveys over the phone. Wo made a second call to all remaining nonrespondents one to 
two weeks later. 



Follow-up procedures for SERVE's "School Safety and Emergency Preparedness: Reducing 
School Violence" training were slightly different. Because the school year for most of these 
participants ended earlier than in much of the rest of the country, they were not available for a second 
follow-up phone call. Therefore, we mailed postcards to their schools to encourage them to complete 
the survey. 
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SCHOOL SAFETY AND EMERGENCY PREPAREDNESS: REDUCING 
SCHOOL VIOLENCE TRAINING 



Participant Survey 

l. Please indicate the type of agency for which you work: (CIRCLE ONE) 



School I 

School district 2 

Intermediate education agency or BOCES 3 

State education agency 4 

Institution of higher education 5 

Other (SPECIFY). 6 



2. Present Title 



3. State 



4. Did you attend the training on School Safety and Emergency Preparedness: Reducing School 
Violence conducted by the SouthEastera Regional Vision For Education (SERVE) in early 
1994? (CIRCLE ONE) 



Yes 1 

No (Skip to end) 2 

Can't remember (Skip to end) 3 

5. If you attended the Reducing School Violence training, how would you rate the quality of the 
training? (CIRCLE ONE) 

a. Very high I 

b. Somewhat high 2 

c. Not very high 3 

d. Not high ataU 4 

5a. How would you rate the quality of the materials you received? (CIRCLE ONE) 

a. Very high 1 

b. Somewhat high 2 

c. Not very high 3 

d. Not high at all , 4 
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5h. Please answer the following hy placing an X in the appropriate hox for each question. 



1 


Very 


Some 
what 


Not 
very 


Not 
at all 


Can't 
remember 


Can't 
judge 


As far as you can judge, how accurate 
and complete was the content of the 
Reducing School Violence training and 
materials? 














How useful was the information to you? 















6. To what extent did this information affect your personal knowledge, understanding, and work 
related to this topic? Did the information: (PLACE AN X IN THE APPROPRIATE BOX 
FOR EACH QUESTION) 





Great 
extent 


Some 
extent 


Very 
little 


Not at all 


help keep you aware of developments and 
activities in education? 










help or encourage you to begin doing some 
new things on your job? 










give you resources for helping others? 










convince you to stop doing something that you 
had been doing on your job? 










Other (Specify) 








= — i 



7. Did you pass the' information or materials from the Reducing School Violence training on to 
anyone inside your school or agency? 



Yes I 

No (SklptoQ7b) 2 

7a. Are they using the information you provided? (CIRCLE ONE) 

Yes 1 

No 2 

Don't know 3 
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7b. To what extent did this information influence your schooLor agency's knowledge, 

understanding, and work related to this topic? Did the information: (PLACE AN X EN THE 
APPROPRIATE BOX FOR EACH QUESTION) 





Great 
extent 


Some 
extent 


Very 
little 


Not at 
all 


Don't 
know 


.support a policy, procedure, or practice 
that was being discussed or debated? 












help generate awareness of the topic in 
your agency? 












help in initiating a new program, policy 
or practice? 












help to determine whether to terminate a 
program, policy, practice? 












Other (Specify) 













8. Did you pass the information or materials from the Reducing School Violence training on to 
anyone outside your school or agency? 



Yes 1 

No (Skip to Q9) 2 

8a. Are they using the information provided in the materials? (CIRCLE ONE) 

Yes : 1 

No 2 

Don't know ' 3 

9. In terms of usefulness, how did the Reducing School Violence training and the accompanying 
materials compare to other training or experiences you have attended that were offered by 
other staff development or assistance providers? (CIRCLE ONE) 

a. Much more useful 1 

b. Somewhat more useful 2 

c. Equally as useful 3 

d. Somewhat less useful 4 

e. Much less useful 5 
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10. Did you consider the Reducing School Violence training to he a good use of your time? 
(CIRCLE ONE) 

a. Yes, it provided the right amount of information on (he 

topic in an appropriate amount of time 1 

h. Yes, it was worth attending but the training only scratched 

the surface. I need more information, interaction, or follow-up 

to really help me or my organization . . 2 

c. No, it was not worth attending because it only scratched the 
surface. I would need more information, interaction, or follow-up 

to really help me or my organization 3 

d. No, because the quality of the activity was not very high 4 

e. Other (PLEASE SPECIFY) ... 5 



1 1 . Based on your experience, please indicate whether you thought the Reducing School Violence 
training was unique when compared to other training you have attended. (CIRCLE ONE) 

a. It was unique in some way (PLEASE ELABORATE) 1 



b. It was not unique, but was of high quality compared 

to other training I have attended .... 2 

c. It was not unique in any way, nor was it of high quality 3 

d. I really cannot judge 4 
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12. What is your relationship with the Southeastern Regional Vision tor Education (SERVE) and 
did it change after you attended the Reducing School Violence training? (CIRCLE ONE) 

a. Before attending the Reducing School Violence training, I was not 
familiar with SERVE and 1 don't expect to contact them for 

additional materials or assistance 1 

b. Before attending the Reducing School Violence training, I was not 
familiar with SERVE T but attending the training encourages me 

to contact SERVE for additional materials or assistance 2 

c. The Reducing School Violence training is one of many services (e.g., 
publications, direct assistance) 1 have received from SERVE, but I 

have not contacted them for additional materials or assistance 3 

d. The Reducing School Violence training is one of many services (e.g., 
publications, direct assistance) I have received from SERVE, and I 

contact them when I need additional materials or assistance 4 

e. Other (SPECIFY) . . 5 



13. We would be interested tn any examples you might like to share about your personal use or 
your agency's use of the information provided in the School Safety and Emergency 
Preparedness: Reducing School Violence training. Please discuss these below: 



Thank you for your help! Please return the completed survey in the enclosed postage-paid envelope 

to: 

Ms. LisaWeiner 
Policy Studies Associates, Inc. 
1718 Connecticut Ave., NW 
Suite 400 . 
Washington, DC 200O9 
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